System and Method for Synthetically Filling Ladar Frames Based on Prior Ladar Return Data

ABSTRACT

Systems and methods are disclosed where a ladar system synthetically fills a ladar frame. A ladar transmitter can employ compressive sensing to interrogate a subset of range points in a field of view. Returns from this subset of range points correspond to a sparse ladar frame, and interpolation can be performed on these returns to synthetically fill the ladar frame.

CROSS-REFERENCE AND PRIORITY CLAIM TO RELATED PATENT APPLICATIONS

This patent application claims priority to U.S. provisional patentapplication Ser. No. 62/750,540, filed Oct. 25, 2018, and entitled“Adaptive Control of Ladar Systems Using Spatial Index of Prior LadarReturn Data”, the entire disclosure of which is incorporated herein byreference.

This patent application also claims priority to U.S. provisional patentapplication Ser. No. 62/805,781, filed Feb. 14, 2019, and entitled“Adaptive Control of Ladar Systems Using Spatial Index of Prior LadarReturn Data”, the entire disclosure of which is incorporated herein byreference.

This patent application is also related to (1) U.S. patent applicationSer. No. ______, filed this same day, and entitled “Adaptive Control ofLadar Systems Using Spatial Index of Prior Ladar Return Data” (saidapplication being identified by Thompson Coburn Attorney Docket Number56976-18508), (2) U.S. patent application Ser. No. ______, filed thissame day, and entitled “Adaptive Control of Ladar Shot Energy UsingSpatial Index of Prior Ladar Return Data” (said application beingidentified by Thompson Coburn Attorney Docket Number 56976-185809), (3)U.S. patent application Ser. No. ______, filed this same day, andentitled “Adaptive Ladar Receiver Control Using Spatial Index of PriorLadar Return Data” (said application being identified by Thompson CoburnAttorney Docket Number 56976-185810), (4) U.S. patent application Ser.No. ______, filed this same day, and entitled “Adaptive Control of LadarShot Selection Using Spatial Index of Prior Ladar Return Data” (saidapplication being identified by Thompson Coburn Attorney Docket Number56976-185811), and (5) U.S. patent application Ser. No. ______, filedthis same day, and entitled “Adaptive Control of Ladar System CameraUsing Spatial Index of Prior Ladar Return Data” (said application beingidentified by Thompson Coburn Attorney Docket Number 56976-185812), theentire disclosures of each of which are incorporated herein byreference.

INTRODUCTION

Safe autonomy in vehicles, whether airborne, ground, or sea-based,relies on rapid precision, characterization of, and rapid response to,dynamic obstacles. Ladar systems are commonly used for detecting suchobstacles. As used herein, the term “ladar” refers to and encompassesany of laser radar, laser detection and ranging, and light detection andranging (“lidar”).

However, it is sometimes the case that artifacts and noise may bepresent in the ladar return data and ladar images used for objectdetection with autonomous vehicles. Such artifacts and noise can hamperthe analysis operations that are performed on the ladar return data. Forexample, machine learning is often used to train an image classifierfrom a set of ladar images so that the trained image classifier canaccurately detect and classify different types of objects in images.However, the presence of artifacts and noise in the ladar images cancorrupt the training process and/or the classification process, whichcan lead to a risk of misclassification during vehicle operation.

The inventors believe that some of the artifacts and noise present inladar return data arise from non-uniform illumination of the field ofview by the ladar system. More particular, the ladar system mayilluminate some portions of the field of view with ladar pulses moreheavily than other portions of the field of view.

Accordingly, in an example embodiment, the inventors disclose a ladarsystem that adapts shot energy for the ladar transmitter as a functionof prior ladar return data so that the ladar system can achieve a moreuniform illumination (or smoother illumination) of nearby parts of thefield of view. Accordingly, the ladar transmitter may adjust its shotenergy on a shot-by-shot basis for interrogating range points that arenear each other in the field of view. It should be understood that thegoal of increasing the uniformity or smoothness of illumination by theladar transmitter over a region of nearby portions of the field of viewdoes not require the ladar transmitter to produce equal illumination foreach range point in that region. Instead, it should be understood thatincreased uniformity or smoothness is a soft term that relates to agradient of intensities that is sufficiently mild so as to not undulyerode the performance of object classification algorithms (many of whichmay be powered by machine learning techniques, edge detection processes,bounding boxes, and others).

Furthermore, a number of factors can make the goal of increasing theuniformity illumination technically challenging. First, the propagationcharacteristics of the environment between the ladar transmitter and thetargeted range point may be variable. For example, the amount ofatmospheric attenuation often varies, which can yield fluctuations inthe energy levels of the ladar return data. Second, the energydischarged by the ladar transmitter can vary from shot-to-shot. This canbe especially true for lasers which have adjustable pulse energy, suchas with many fiber lasers. Third, the angular sensitivity of the ladarreceiver may vary. This can be most notable in bistatic operations wherethe ladar system scans on transmit but not on receive. In such a case,the ladar return data detected by the ladar receiver may exhibit angularvariation, whether it be a focal plane array or a non-imaging system.

Accordingly, for example embodiments, the inventors believe that ladarsystem adaptation should be based on the observed behavior of the ladarreturns. Because current lasers operate at a fast re-fire rate (e.g.,100,000 to 3,000,000 shots per second is typical), this means that usingladar return data to adapt the ladar system with low latency is acomputationally-challenging task. For example, a ladar system might scana 300 m swath, in which case the last ladar return arrives at the ladarreceiver around 2 microseconds (us) after ladar pulse launch from theladar transmitter. If there are 300,000 ladar pulse shots per second,this leaves only about 1.3 us to detect ladar pulse returns, estimatethe shot energy desired for the next ladar pulse shot (to more uniformlyilluminate a region of the field of view), and then prime the laser pumpfor that next ladar pulse shot (if it is desired for the ladar system tohave the ability to adapt shot energy on a shot-by-shot basis). Giventhat many pulsed lasers have bandwidths close to 1 GHz, this is adaunting computational task as well as a daunting data management task.

In order to provide a solution to this problem in the art, the inventorsdisclose example embodiments where the ladar system stores data about aplurality of ladar returns from prior ladar pulse shots in a spatialindex that associates ladar return data with corresponding locations ina coordinate space to which the ladar return data pertain. This spatialindex can then be accessed by a processor to retrieve ladar return datafor locations in the coordinate space that are near a range point to betargeted by the ladar system with a new ladar pulse shot. This nearbyprior ladar return data can then be analyzed by the ladar system to helpdefine a shot energy for use by the ladar system with respect to the newladar pulse shot. Accordingly, the shot energy for ladar pulse shots canbe adaptively controlled to achieve desired level of illumination forthe range points within a defined vicinity of each other in thecoordinate space (e.g., a more uniform level of illumination).

The spatial index of prior ladar return data can take the form of a treestructure that pre-indexes the prior ladar return data, where the treestructure has a root node, a plurality of branches (with associatedbranch nodes), and a plurality of leaf nodes, wherein the leaf nodesassociate the ladar return data with corresponding locations in thecoordinate space. Through such a tree structure, rapid lookups to findthe prior ladar return data for locations within a defined vicinity ofthe new ladar pulse shot can be performed. As an example, the treestructure can be a quad tree index, where 4 leaf nodes are linked to acommon first level branch, 4 first level branches are linked to a commonsecond level branch, and so on until the root node is reached, and wherethe branch topology is selected to reflect spatial proximity. Thecomputational complexity of performing data retrieval from an examplequad tree index is 2 p log₄(m), where m is the number of rows andcolumns in the coordinate space grid, and where p is the number of priorladar returns to retrieve and inspect. In contrast, withoutpre-indexing, the complexity, including memory fetches, to retrieve pprior ladar returns would be approximately pm². For a ¼ Mega-pixel ladarframe, the cost savings (for a given p) of using a quad tree index topre-index the prior ladar return data is on the order of 55 times. Thesavings are even more dramatic, since the dominating component of thecomputational complexity for quad tree indexes is p rather than m, andwhere p varies with the search radius R used in an example embodiment togovern the locations in the coordinate space are deemed to be within thedefined vicinity of the new ladar pulse shot.

While an example embodiment uses the spatial index of prior return ladarto adaptively control shot energy, it should be understood that in otherexample embodiments parameters of the ladar system other than shotenergy can be adaptively controlled based on the spatial index of priorreturn ladar. For example, the detector and/or comparison thresholdsused by the ladar receiver can be adaptively controlled using thespatial index. As another example, shot selection itself can beadaptively controlled using the spatial index. For example, with respectto adaptive shot selection, the techniques used for the adaptivemodification of the spatial index can be used to preemptively reduce thenumber of required shots in the shot selection stage. As an example, thesystem can use super-resolution to interpolate by dropping random shots.As another example, the system can use optical flow techniques incombination with spatial indexes to adapt shot selection.

These and other features and advantages of the present invention will bedescribed hereinafter to those having ordinary skill in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows an example ladar system for use in an example embodiment.

FIG. 1B shows an example process flow that leverages a spatial index ofprior ladar return data to define a parameter value for use by the ladarsystem with respect to a new ladar pulse shot.

FIG. 2A shows an example quad tree index structure for associating priorladar pulse return data with locations in a coordinate space from whichthe prior ladar pulse return data was obtained.

FIG. 2B shows an example where various leaf nodes of a quad tree indexare populated with prior ladar return data.

FIGS. 2C and 2D show example tree views of the quad tree index of FIG.2B.

FIG. 3A shows an example process flow that leverages a spatial index ofprior ladar return data to define a shot energy with respect to a newladar pulse shot.

FIG. 3B shows example pseudo-code for the portions of FIG. 3A relatingto searching the spatial index to find nearby prior ladar return data.

FIG. 4 shows an example process flow that describes how interpolationcan be performed on prior ladar return data from the spatial index tocompute a desired shot energy for a new shot.

FIGS. 5A and 5B shows example process flows that leverage a spatialindex of prior ladar return data to adjust a control setting for a ladarreceiver with respect to processing a ladar return with respect to a newladar pulse shot.

FIG. 6 shows an example process flow that leverages a spatial index ofprior ladar return data to adjust shot selection for the ladar system.

FIG. 7 shows an example system diagram for a ladar system that leveragesa spatial index of prior ladar return data to control various parametervalues with respect to a new ladar pulse shot.

FIGS. 8 and 9 show examples of ladar images produced by non-adaptive andadaptive ladar systems respectively.

FIGS. 10A and 10B show example use cases for leveraging a spatial indexof prior ladar return data to better detect objects in dark regions ofthe field of view.

FIG. 11 shows an example use case for leveraging a spatial index ofprior ladar return data to detect and adapt to interference.

FIG. 12A shows an example use case for leveraging a spatial index ofprior ladar return data to detect and adapt to the presence of nearbycamera phones.

FIG. 12B is an example plot showing how camera damage can be affected bydistance and shot energy.

FIG. 13A shows an example process for synthetically filling a ladarframe from a sparse array of ladar return data.

FIG. 13B shows examples of images derived from a sparse ladar array anda synthetically-filled ladar array.

FIG. 14 shows an example process for synthetically filling ladar framesfrom using a rolling shutter technique.

FIG. 15 shows an example scene that illustrates an optical flow conceptwith respect to moving vehicles.

FIG. 16 shows an example process flow where the spatial index of priorreturn data can be used in combination with optical flows to adaptivelycontrol shot selection.

FIG. 17 shows an example ladar system that includes a camera.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1A shows an example ladar system 100. The ladar system 100comprises a ladar transmitter 102, a ladar receiver 104, and a controlsystem 106. The ladar transmitter 102 is configured to generate andtransmit ladar pulses 110 into the environment toward targeted rangepoints (e.g., range point 112). The ladar receiver 104 is configured toreceive and detect incident light that may include ladar pulsereflections 114 from targeted range points such as range point 112. Thecontrol system 106 can be configured to control how its correspondingladar transmitter 102 and ladar receiver 104 operate. Examples ofsuitable ladar systems 100 are disclosed and described in greater detailin U.S. Pat. App. Pubs. 2016/0047895, 2017/0242106, 2017/0307876,2018/0238998, and U.S. patent application Ser. No. 16/106,350, entitled“Intelligent Ladar System with Low Latency Motion Planning Updates”,filed Aug. 21, 2018; the entire disclosures of each of which areincorporated herein by reference. For example, the ladar system 100 mayemploy a ladar transmitter 102 (as described in the above-referenced andincorporated patent applications) that includes scanning mirrors anduses a range point down selection algorithm to support pre-scancompression (which can be referred herein to as “compressive sensing” or“compressive scanning”). Such an embodiment may also include anenvironmental sensing system that provides environmental scene data tothe ladar transmitter to support the range point down selection. Throughthe use of pre-scan compression, such a ladar transmitter can bettermanage shot rate and digital input/output bandwidth through intelligentrange point target selection. Furthermore, because the detection andimage quality for a ladar system varies roughly as the square root ofthe number of pulses used per point cloud, this means that reducing therequired number of communication pulses via the compressive sensingenhances the signal to noise ratio (SNR), enabling robust pulsecollision avoidance without greatly reducing detection range or positionaccuracy. While these referenced and incorporated patent applicationsdescribe example embodiments for ladar systems 100, it shouldnevertheless be understood that practitioners may choose to implementthe ladar systems 100 differently than as disclosed in these referencedand incorporated patent applications.

FIG. 1B shows an example process flow for execution by a processorwithin control system 106. This process flow leverages a spatial indexof prior ladar return data to define a parameter value for use by theladar system 100 with respect to a new ladar pulse shot.

At step 150, the processor determines whether a new ladar pulse shot isto be taken. As explained in the above-referenced and incorporatedpatent applications, the processor can identify new ladar pulse shotsbased on a shot list that includes an ordered listing of ladar pulseshots. Each ladar pulse shot can be identified by the coordinates of therange point to be targeted by that ladar pulse shot. For example, thesecoordinates can be identified by x,y values in terms of elevation andazimuth in a field of view coordinate space for the ladar transmitter102. Rectangular to polar conversion can be applied if necessary totranslate coordinates from one system to another, folding in anadditional parameter, mainly range. Such range point coordinates can bereferred to as pixel locations for the ladar system 100. The range pointcoordinates for the new ladar pulse shot are then identified by theprocessor at step 152.

Next, at step 154, the processor searches a spatial index 160 to findprior ladar return data for range points that are near the targetlocation identified at step 152. The spatial index 160 associates returndata from prior ladar pulse shots with the locations of the range pointstargeted by those prior ladar pulse shots. Accordingly, step 154 candefine a vicinity around the identified range point location from step152 to establish the zone of range point locations that will qualify asbeing “nearby” the targeted range point. As an example, the nearbyvicinity can be defined in terms of a radius around the targeted rangepoint location (e.g., where such a radius value can be expressed as acount of pixels or some other suitable unit). Prior ladar return datathat is associated with a location within such a defined vicinity isthen located within the spatial index 160 as a result of step 154.

At step 156, the processor processes and analyzes the nearby prior ladarreturn data from step 154. The analysis that is performed at step 156can vary based on the type of control that a practitioner wants toemploy over the ladar system. For example, if a practitioner wants toincrease the uniformity of illumination of nearby range points by theladar transmitter 102, step 156 can include an analysis where theintensity of nearby ladar returns can be analyzed. As another example,if a practitioner wants to exercise adaptive control over shotselection, an absence or sparseness of returns analyzed at step 156 canbe construed by the processor as an indication that the region hasnothing of interest, in which case shot selection can be adjusted toadopt a sparser sampling (e.g., if the ladar system is pointing into theopen sky, it may be desirable to only sparsely sample such empty space).Similarly, if the analysis of the returns reveals a relatively uniformintensity, this may be indicative of an amorphous heterogeneousbackground (e.g., road, dirt, grass, etc.), which the processor mayconstrue as indicating a relaxing of that region's scan priority infavor of more intensity dynamic regions. The returns can also beanalyzed for changes in texture, using techniques such as Markovianfield parameter estimation. This allows the user to segment the imagebased not on edges or range but rather on spatial stochastic propertiesof surfaces, enabling for example a lidar-only characterization of grassversus asphalt.

At step 158, the processor applies the analysis from step 156 to definea value for a parameter used by the ladar system 100 with respect to thenew ladar pulse shot. The nature of step 158 can also vary based on thetype of control that a practitioner wants to employ over the ladarsystem. For example, if a practitioner wants to increase the uniformityof illumination of nearby range points by the ladar transmitter 102,step 158 can define a shot energy for the new ladar pulse shot so thatthe ladar pulse shot would illuminate the targeted range point with anenergy amount that is derived from the intensities of the prior ladarreturns from nearby range points. As another example, step 158 caninvolve defining values used by the ladar receiver 104 with respect todetection/comparison thresholds. As yet another example, step 158 caninvolve tagging the new ladar pulse shot as a shot not to be taken or tobe deferred if the prior ladar pulse return data for nearby range pointsindicates there may be an occlusion or other factor that would make thereturn data unreliable. Additional details about such examples arediscussed below with reference to example embodiments.

FIG. 2A depicts an example tree structure 200 for a spatial index 160.As can be seen, the tree structure 200 can be hierarchical with multiplelevels. In the example of FIG. 2A, the tree structure 200 includes afirst level 202, a second level 204, a third level 206, and a fourthlevel 208. In this example, the fourth level 208 will be the lowestlevel and will define a number of cells 218 that subdivide the field ofview for the ladar system 100 into different spatial locations at thelowest level of granularity with respect to the subject tree structure200. Each cell 218 can be characterized as a pixel location in the ladarsystem's field of view, and can be indexed by an x,y coordinate pairthat would correspond to elevation/azimuth values in the field of view.Populating each cell 218 can be data that describes a prior ladar pulsereturn from a range point at that coordinate. Examples of ladar returndata that can be used to populate cells 218 can include range andintensity information for the range point location corresponding to thesubject ladar return. But, the ladar return data may also oralternatively include additional information such as features of thedetected return (e.g., target shape in range), noise level data for theladar return from the subject range point location, pulse shape data forthe ladar return from the subject range point location, multiple returns(when there are multiple objects at a given azimuth and elevation beamat two or more distinct range positions in depth—where this is, forexample, a frequent occurrence at long range on busy roads). It may alsobe desirable to record the exact time of collection of each pulse return(which may be measured with granularity that is orders of magnitudetighter than the frame time). Time of collection, more precisely theexact time that each pulse is launched, can be presented as a timestamp, e.g., a time stamp data tag. Such a time stamp can be useful forfusing data from other sensors, such as radar, sonar etc. Other examplesof data that can reside in the leaf nodes can include camera data (redgreen blue, polarization), map data (such as terrain height, and terrainclassification (e.g., building, foliage, dirt etc.)), and/or velocity ofobjects detected using intershot range detection.

In the example of FIG. 2A, the tree structure 200 is arranged as a quadtree index. Accordingly, each higher layer level operates to group 4cells within the immediate lower layer level. Thus, each cell 216 inlevel 206 operates to group a 2×2 array of cells 218 from level 208.Such grouping can start from the upper left corner and work its wayacross and down the array. However, it should be understood that thegrouping could start from other origins (e.g., upper left corner, lowerleft corner, lower right corner, mid-point, etc.). Given that theexample of FIG. 2A shows an 8×8 array at lowest level 208, this meansthat level 206 has a 4×4 array of cells 216, level 204 has a 2×2 arrayof cells 214, and the highest layer level 202 is a single cell 212(e.g., the full frame of the field of view). Level 202/cell 212 can alsobe referred to as the root node since it references the entire field ofview for a subject frame. Each cell within the tree structure 200 can bereferred to as a node; and the cells 218 within lowest level 208 can bereferred to as leaf nodes, while the cells in the higher layer levelsabove the leaf nodes and under the root node can be referred to asbranch nodes (or standard nodes or non-leaf nodes). Branches link thenodes together.

Through the hierarchical spatial organization of the tree structure 200shown by FIG. 2A, efficient lookups can be performed to locate the rangepoint cells that are nearby a range point cell corresponding to a newladar pulse shot. Each node in the tree structure 200 can encode thestatus of its descendants. Thus, if there are any active leaf nodesunder a given node/branch, then that node/branch can also be encoded asan “active” node/branch. A leaf node can be deemed “active” if it holdsladar return data that is considered to have been recently obtained, asdiscussed in greater detail below. Such status encoding can helpcontribute to fast lookups in the tree structure 200 because it canpre-index which branches have active ladar return data below them.Furthermore, each branch node in the tree structure can also identifythe boundaries (e.g., 2D boundary; which may take the form of arectangle described by 4 numerical values—the upper left and lower rightcorner coordinates, for example) within which all of its descendantnodes lie. Such boundary encoding can also help support fast lookups ofactive leaf nodes within a specified proximity of a point of interest.Even in worst case scenarios where the range point locationcorresponding to a new ladar pulse shot is located on a cell 218 thatlies on a corner boundary of a high level cell (such as the corner ofone of the cells 214 in level 204), there will be a fixed maximum numberof lookups that would need to performed derived from the relationship of2p log₄(m) discussed above to find all cells 218 within a defined radiusR of the subject cell. Because of the spatial organization of the cells218 in the tree structure 200, nearby cells can be located by traversingthe tree structure 200 to higher layer levels as needed, shifting to anadjacent cell, and then traveling back down to low level cells 218 underthe adjacent cell as needed.

With the example of FIG. 2A, the tree structure 200 will have a rootnode 202/212 that then spatially subdivides the coordinate space viabranch nodes 214 and 216 in levels 204 and 206 down to spatially groupedleaf nodes 218 in the lowest level 218. As noted, the prior ladar returndata can be stored in these leaf nodes. The spatial extent of a leafnode 218 is typically bounded below by the angular resolution of theladar system. So, for example, if the beam resolution is 0.1 degrees, apractitioner can set the resolution to 0.1 degrees, but may choose ithigher, perhaps 1 degree. The trade here is on computation resourcesspent accessing and comparing data within a cell/node, versus comparingthe data between nodes. The trade can also be on the spatial variationof intensity; the faster it changes the smaller the angular extent. Asan example, we may elect to use as spatial extent the width of a road ata given reference range depth. This would be appropriate when theprimary search would be for discerning texture changes.

Also, while the example of FIG. 2A shows the lowest level 208 comprisingan 8×8 array of cells 218, it should be understood that in manyembodiments, there will be much larger numbers of cells 218 in thelowest level 218. For example, in a 1 million pixel (˜2²⁰) array forladar frames, the tree depth is log₄4¹⁰=10, and the lowest level 208 canbe a 1000×1000 array of cells 218. Also, while the example of FIG. 2Ashows the tree structure 200 as a quad tree index, it should beunderstood that other groupings could be used, such as an octree index.An octree index may be desirable if the tree structure seeks to embed3-dimensional (3D) spatial information directly, where depth isexplicitly taken into consideration in its indexing of nodes, as opposedto a quad tree index where depth can reside inside a leaf node asopposed to living in the tree index map itself. Thus, while the quadtree index can spatially index the ladar return data in a 2D pixelcoordinate space, an octree index can spatially index the ladar returndata in a 3D voxel coordinate space. Indexing voxels can involvecreating 2×2×2 cubes as leaf node pairings and where the additionaldimension corresponds to the range for the subject range point. With anoctree embodiment, a practitioner may then choose to also take anexpected range for a targeted range point into consideration whendetermining which voxels are deemed to be in the vicinity of thetargeted range point.

In an example embodiment, the root node 202/212 can address the entirecorpus of prior ladar return data indexed by the tree structure 200.This corpus can correspond to a single frame of prior ladar return data,multiple frames of prior ladar return data over a defined time period,or other groupings of prior ladar return data. It should be understoodthat the tree structure 200 provides a scheme for indexing cells. Thecontent in the leaf node cells can be whatever a practitioner decides.If multiple frames reside in a single tree structure 200, then it may bedesirable for the leaf nodes to provide an age index, e.g. how manyframes ago it was shot. This could be generalized to an absolute timestamp, if desired. If age is above a specified user threshold,leaf/nodes are then determined “stale”, and recursively deactivated. Inexample embodiments where the ladar system employs compressivesensing/scanning, it should be understood that a frame can be a fluidframe given that the same shots will presumably not be collected fromframe-to-frame. This stands in contrast to non-agile ladar systems wherea frame is a fixed set of range points, fired repeatedly andinvariantly. With an agile ladar system that employs compressivesensing/scanning, a frame can be viewed as a collection of range pointshots that loosely encompasses a field of view, and a subsequent framerevisits said view, albeit in a modified manner (e.g., different rangepoints being targeted with shots).

Recall, by definition, each potential leaf node can be associated with astatus identifier that identifies whether that leaf node is considered“active” or not. Deactivating leafs nodes in general may also lead todeactivating branches and branch nodes leading to that deactivated leafnode. Indeed, this is the power of a quad tree index when used with anagile ladar system that employs compressive sensing/scanning because avast majority of leaf nodes are never searched since they are notactive. The processor tracks which leaf nodes should be defined asactive based on the freshness or staleness of their respective ladarreturn data. For instance, the system can time tag returns so that aleaf node can be denoted as “stale” if all time stamps are old, and“fresh” if it contains fresh ladar return data. The processor can thentrack which leaf nodes to search across based on the freshness orstaleness of their respective ladar return data, by inspectingassociated (currently) active leaf nodes. If the leaf node is inactive,one can simply write over the stale ladar return data still stored bythat leaf node the next time that the ladar system receives return datafor the range point corresponding to that leaf node.

A practitioner may employ a constraint on how long the tree structure200 holds prior ladar return data; a practitioner may not want “stale”ladar return data to influence the adaptive control of the ladar systemon the assumption that conditions may have changed since the time that“stale” ladar return data was captured relative to the current time. Theprecise time durations or similar measures employed by the ladar system100 to effectively flush itself of “stale” ladar return data can varybased on the needs and desires of a practitioner. For example, in someexample embodiments, it may be desirable to flush the tree structure ofprior ladar return data after each ladar frame is generated. In otherexample embodiments, it may be desirable to flush the tree structure ofprior ladar return data after a defined sequence of ladar frames hasbeen generated (e.g., holding the ladar return data for a given framefor a duration of, say, 3 ladar frames such that the returns for theoldest of the 3 ladar frames is dropped when a new ladar frame isstarted). In other example embodiments, it may be desirable to managethe ladar return data on a shot-by-shot basis where ladar return data isheld in the tree structure for X number of shots. It is also possible tohold the tree structure 200 for an amount of time which is datadependent. For example, a practitioner may elect to reset the treestructure 200 whenever the entire set of possible branches have beenpopulated, i.e. when the number of leaf nodes is 4^(n) where n is thenumber of branching levels (depth) in the tree. Alternatively, the treestructure 200 may be reset when the stochastic structure of the leafcontents shifts, as may be measured using principal component analysis.

Further still, any of a number of mechanisms can be employed forimplementing control over the freshness of data in the tree structure.For example, the mean time between updates can be a criterion fordeactivation, or the intensity variance, which if too large may indicatethat the returns are simply receiver noise. A complete purge can beperformed each frame or arbitrarily, but if we are indexing the previousn frames, a partial purging can be done each frame to remove all old(stale) leaf nodes (where “old” here means exceeding some timethreshold). Otherwise if we don't remove or deactivate “stale” leafnodes, they would be included in subsequent searches. A practitioner maydecide not to purge from memory inactive leaf nodes of previously storedladar return data. In so doing, the practitioner can both streamlinesearch (deactivated nodes are removed from the tree reducing searchcomplexity) while simultaneously storing “stale” data that may resurfaceas useful later on. Suppose, for example, that a ladar-equipped cardrives into a tunnel. The environment suddenly changes dramatically,what was sky is now suddenly filled with returns from the tunnelceiling, etc. This sudden change in the environment may quickly resultin useful data being deemed “stale”. When the vehicle exits the tunnel,the environment will revert back to what it was before. Far better then,to “resurrect” the old “stale” data and update it, than start fromscratch. For example the road surface, trees or lack thereof, weatherconditions which reflect return intensity etc. will likely be highlycorrelated with conditions extant before entering the tunnel.

The power of quad tree indexing with respect to adaptive ladar systemcontrol becomes even stronger when the ladar return data is sparse,which is the general case for an intelligent and agile dynamic rangepoint ladar system that employs compressive sensing/scanning. In such acase, as shown by FIGS. 2B-2D discussed below, the quad tree index andlookups are greatly simplified, in a way that direct search is not.

FIG. 2B shows an example where various leaf nodes of a quad tree indexare populated with prior ladar return data. In an example embodimentwhere the ladar transmitter 102 employs compressive sensing/scanning,the ladar frames detected by the ladar system 100 can be characterizedas sparse arrays because only a subset of the range points within theframe's field of view will have been targeted with ladar pulses. Thus,in example embodiments, each ladar frame sensed by the ladar system 100may have only a relatively small number of range points from which ladarreflections/returns are received. This means that the quad tree indexwill have a relatively sparse number of leaf nodes that are populatedwith prior ladar return data for a given frame. An example of this isshown by FIG. 2B where the 8×8 array of leaf nodes in level 208 ispopulated with only 4 prior ladar returns (as denoted by the solidlyshaded cells in FIG. 2B). If a new ladar pulse shot is to target therange point shown in FIG. 2B by cross-hatching, the ladar system 100will then apply the FIG. 1B process flow to find the leaf nodes that arenearby the to-be-targeted range point. Depending on the radius R used todefine the vicinity around the targeted range point that qualifies asnearby, there will be a maximum of 4 potential ladar returns to accessand consider at steps 154 and 156 of the FIG. 1B process flow. We willstep through this training set selection process, and streamlining(pruning) of branches below.

In the example of FIGS. 2A-2D, a labeling convention can be applied tothe tree structure such that the 4 cells 214 in level 204 can be labeledA, B, C, and D with cell A being in the upper left corner and then theother cells being labeled in alphabetical order in a top down and leftto right direction from the upper left corner. This means that the upperright cell will be labeled B, the lower left cell will be labeled C, andthe lower right cell will be labeled D. Moving down to the next level206, the cells 216 will be labeled by pairs of letters where the firstcharacter in the pair sequence is inherited from the spatially relevantcell 214 in the higher level 204 and where the second character in thepair sequence uses the same rotation of ABCD values. Accordingly, theupper left cell 216 in level 206 would be labeled AA, while the lowerright cell 216 in level 206 would be labeled DD. Moving down to level208, the cells 218 will be labeled by tuples of letters, where the firstcharacter in the sequence is inherited from the spatially relevant cell214 in level 204, where the second character in the sequence isinherited from the spatially relevant cell 216 in level 206, and wherethe third character in the pair sequence uses the same rotation of ABCDvalues. Accordingly, the upper left cell 218 in level 208 would belabeled AAA, while the lower left cell 218 in level 208 would be labeledCCC. With this labeling convention, FIG. 2B shows that leaf nodes ADA,ADD, BCA, and CDD each contain prior ladar return data. FIG. 2B alsoshows that the next range point to be targeted with a ladar pulse shotcorresponds to leaf node AAD. FIG. 2C provides a tree view of this quadtree index, which shows the spatial relationship arising from thebranching in levels 204 and 206 down to the leaf nodes 218 in level 208.From FIG. 2C, it can be seen that leaf nodes ADA and ADD are both underbranch node A together with leaf node AAD that is to be targeted with anew ladar pulse shot. Furthermore, leaf nodes ADA and ADD are also bothunder branch node AD (although leaf node AAD is not under branch nodeAD, and is instead under branch node AA). Furthermore, leaf node BCAfalls under branch nodes BC and B respectively; while leaf node CDDfalls under branch nodes CD and C respectively. It should be understoodthat the labeling of FIGS. 2B and 2C can be conceptual; we shall now seethat only a small section of the cells need be used and that the depthof branches can be vastly reduced if desired by a practitioner.

But before we begin, note that in a “pure play” quad tree index, such asis defined here in an example embodiment, the branch nodes only provide(i) pointers to their (first generation) children (where such childrenmay be leaf or branch nodes), and (ii) the bounding box corners thatdefine the contour which encompasses all leaf nodes that are descendantsof the subject branch node. The role of the bounding box coordinates isthat it allows the quad tree index to be searched to determine if agiven leaf node is within a distance R of any leaf in the tree, quicklydiscarding the need for any further testing below the branch node. Thisis why the logarithm term appears in the complexity search model. Bydesign, the area defined by all branch nodes in the tree are always suchthat their union is the entire ladar field of view.

However, it should be understood that a practitioner may choose to storeadditional information in branch nodes if desired. For example, apractitioner may find it desirable to store data in branch nodes thataggregates all of the return data for active leaf nodes under thatbranch node. In this way, if the search for nearby returns wouldencompass all of the active leaf nodes under a given branch node, thenthe lookup process would need to only access the subject branch node toobtain aggregated return data (rather than requiring additionaltraversal deeper into the tree structure down to the active leaf nodesunder that branch node). In terms of choosing what information toinclude in branch nodes, the trade off of what information (if any) ofthe shot returns to store in branches close to the root will surroundsissues such as pre-processing, memory, versus recurring execution timeper query.

We are now poised to discuss an example embodiment for using quad treesin connection with the example of FIG. 2A. Define the x,y origin as theupper left hand corner, growing from top to bottom and left to right.Assume small angles so x,y can be viewed as distance without polarconversion. Each box is assumed to be one unit in angular distance; soAAD, the sample we need training data for, has x=1, y=1. Likewise, thetraining samples have spatial coordinates ADA, x=2, y=2, ADD, x=3, y=3,BCA, x=4, y=2, CDD, x=3, y=7. Next assume the range measured for the newshot, and old shots, are, respectively, (2, 7, 6, 3, 12), and finallythat the search region is R=4.

Now the goal in a quad tree is to have (up to) 4 children for eachbranch node, where the children can themselves be either branch nodes orleaf nodes. Recall we define the bounding box where all the branch nodechildren “live” as well. The boxes can be rectangles so that we keep thesearch geometry simple. FIG. 2D shows a quad tree graph for the data setin FIGS. 2C and 2D including all the data stored at graph nodes forcompleteness.

The root of the tree is 212, which encompasses all data that has beencollected in a frame, or set of frames. The root is the “foundation” ofthe tree and does not per se possess any information, it is just aninitialization step. The first set of nodes, 204, below the tree are allbranch nodes, labeled A, B, C, D as per FIGS. 2A-2C. These nodes containas content, 214, the x,y pairs describing the bounding regions for theaddressable leaf nodes (which are not necessarily active leafs). Forexample, D is the lower right quadrant whose edges are x=4, y=4, andx=7, y=7, as shown with {[ ], [ ] } notation in FIG. 2D. Now we begin byfinding out which branch nodes in layer 204 are within 3 of the new shotAAD. Note that each test requires only two multiplies, for a total of 8multiplies. In our case, ADA, ADD, BCA satisfy this and are added to theelements of the leaf node for AAD. For ease of illustration, we showleaf nodes as squares and branch nodes as circles.

In our case, all leaf nodes are at the second layer in the tree, 206,with entries, 216, that consist of x, y, range triplets (for trainingnodes), and after the search the current node includes the training setpointers as well. Whether or not to delete these after interpolation isfor the user to decide.

It should be evident that FIG. 2D shows a vast reduction in memory aswell as search time as a result of the quad tree approach. The exampleembodiment that we describe here of a quad tree is a root-to-leaf searchscheme. The opposite direction search can also be done, and is useful ifa non-exhaustive, partial, search of training set data is sufficient.Many other variants are possible, such as minimum “Manhattan distance”(sum of absolute errors in x and y), which minimizes multiplications,best m nearest neighbors training etc.

Adaptive Shot Energy Control Using Spatial Index of Prior Ladar ReturnData:

FIG. 3A shows an example process flow that leverages a spatial indexsuch as the quad tree index of FIG. 2D to define a shot energy withrespect to a new ladar pulse shot. With reference to FIG. 3A, theoperations performed above dashed line 350 are focused on finding theladar return data within the spatial index that will be used as thetraining set to control the adaptiveness of the system. FIG. 3B showsexample pseudo-code for the portions of FIG. 3A above dashed line 350.The operations performed below dashed line 350 are focused on how thetraining set is applied to adapt the system (via shot energyinterpolation in this example). In this example, the range points andtheir corresponding coordinates in the field of view can be referred toas pixels. At step 300, a processor selects a new range point to betargeted from the shot list. The associated pixel for this range pointis then identified (e.g., a given azimuth and elevation pair for thatrange point). With reference to the running example from FIGS. 2B-2D,step 300 can identify AAD as the range point to be targeted.

At step 302, the processor selects the maximal search radius, R, that isused to define the vicinity of range points that are considered “nearby”the targeted range point identified at step 300. This value for R can beexpressed as a count of pixels or cells around the targeted range point(e.g., 3 pixels/cells). However, it should be understood that otherunits could be chosen. For purposes of explanation with the runningexample of FIGS. 2B-2D, an R value of 3 pixels/cells will be used. Thevalue of R defines the largest distance away from the targeted rangepoint for which the ladar system will look for previous shots to usewhen evaluating how the ladar system should be adaptively controlledwith respect to the targeted range point. Loosely speaking, in anexample embodiment, the value of R can define the angular extent thatthe intensity in the image is deemed to be “smooth” to quadratic order.

Further still, it should be understood that for some embodiments apractitioner may choose to employ a fixed value of R for all ladar pulseshots; while for other embodiments, a practitioner may choose to adjustthe value for R in certain circumstances. The size of “R” can vary as afunction of the types of objects, the density of objects, and/orlikelihood of presence of objects, that may be known to exist in thefield of view. For example, it may be desirable to controllably adjustthe search radius on a shot-by-shot basis. As another example, thesearch radius can be controlled based on the type of object that isexpected to be targeted by the new ladar pulse shot. Thus, the systemexpects the new ladar pulse shot to target a road surface, the ladarsystem can define the search radius accordingly. Given that the roadsurface is expected to exhibit a relatively smooth return over largedistances, the ladar system may choose to set R to a relatively largevalue. But, if the returns indicate that an object may be present wherethe new ladar pulse is targeted, the system may want to reduce the sizeof R to account for a smaller area that the object might encompass. Thechoice of R presents a tradeoff involving speed, where the smaller Rwill result in faster results. There is another (largely) unrelatedtradeoff, namely the rate of change anticipated for the intensity of thescene. The sky will be unlikely to change in intensity much, while aroad with a reflective sign post will vary quite a bit. Since the systemcan adjust based on the energy expended, the system can largely minimizeany artifacts from intensity fluctuation in the image. After selectingR, the search begins at the tree root (step 304).

At step 306, the processor tests to see if the new range point is withina distance R of the children of the root node (1^(st) generationdescendants which, by design, are branch nodes). Then, for all k suchchildren the search continues as follows. The search drops, at step 308,to the second level of child nodes, which may be either leaf or branchnodes. Of these four nodes the search examines which nodes, if any, arewith radius R of the targeted range point, and labels these as leaf orbranch nodes, numbered as n,m (step 310). Next, one tests whether m>0(step 312); and if so adds these leafs to the list of training cells(step 314). At step 316, the process sets k′ equal to the number thatrepresents all of the branch nodes within n that are no more than R fromthe new range point. Next, at step 318, the processor moves downward inthe tree index by a level and repeats steps 310, 312, 314, and 316 untilthe stack is empty.

With reference to FIGS. 2B-2D and an R value of 3, step 306 would firstresult in finding branch node A because branch A lies within zerodistance of targeted range point AAD (since AAD is inside the upper leftsquare region of the field of view (FOV)). Then (moving left to right),the search would identify that branch nodes B, C lie within a distance 3from AAD, but branch D does not.

The process flow then proceeds to step 308. At step 308, the processormoves upward in the spatial index to the next higher level branch (thegrandchild branch one generation removed from the root). Then, thesenodes are inspected (step 310), which in the FIGS. 2B-2D example revealsthey are all leaf nodes, so step 312 flags “yes”, and step 314 isexecuted. At step 314, leaf nodes with prior ladar return data that arealso within the search radius R are added to the training list. Withreference to FIGS. 2B-2D and an R value of 3, step 314 would result infinding, ADA, ADD, BCA, which are then added to leaf node contents forAAD.

The process flow then proceeds to step 316. At step 316, the processorcontinues the search, 318, down all k′ branch nodes found in 310 untilall leaf nodes have been accessed. With reference to FIGS. 2B-2D and anR value of 3, step 318 would result in a determination that there are nomore leaf nodes that fall within an R value of 3 (see FIG. 2B), i.e.k′=0. Accordingly, the process flow would then advance the process flowto step 320.

At step 320, the processor performs interpolation on the prior ladarreturn data in the active leaf nodes identified at steps 310 and 318 tocompute a shot energy for the new ladar pulse shot that will target thesubject range point. Accordingly, with reference to FIGS. 2B-2D and an Rvalue of 3, step 320 would result in the processor using the ladarreturn data stored by leaf nodes ADA, ADD, and BCA to compute a shotenergy for the ladar pulse shot that will target range point AAD.Examples of techniques for performing this interpolation and shot energycomputation are discussed below.

At step 322, the ladar transmitter 102 transmits the new ladar pulseshot toward the targeted range point, where the ladar transmitter 102controls the laser source so that the new ladar pulse shot has the shotenergy computed at step 320. The laser source of the ladar system canhave adjustable control settings for charge time and average power, andthese parameters can be controlled to define the shot energy. Withreference to FIGS. 2B-2D and a scanning ladar transmitter 102 thatemploys compressive scanning as described in the above-referenced andincorporated patent applications, this operation involves the ladartransmitter 102 firing the new ladar pulse shot with the computed shotenergy toward range point AAD when the scanning mirrors are positionedso that the ladar transmitter 102 targets range point AAD.

At step 324, the ladar receiver 104 then receives and processes a ladarpulse reflection corresponding to the ladar pulse shot fired at step322. As part of this operation, the ladar system 100 is able to extractdata from the reflection such as the intensity of the reflection and therange to the targeted range point. The ladar system can also, ifdesired, extract data from the reflection such as the shape of the pulsereturn, the average noise level, and (when available) polarization. Thisdata can then serve as the ladar pulse return data for the range pointtargeted by step 322. Techniques for receiving and processing ladarpulse reflections are described in the above-referenced and incorporatedpatent applications. With reference to FIGS. 2B-2D, step 324 wouldoperate to generate ladar pulse return data for range point AAD.

At step 326, the processor updates the spatial index based on the newladar pulse return data obtained at step 324. As part of this step, theprocessor can populate a leaf node linked to the subject range pointwith the new ladar pulse return data. With reference to FIGS. 2B-2D,step 326 would operate to populate a leaf node for AAD with the ladarpulse return data received at step 324 for AAD. This operationeffectively adds leaf node AAD to the pool of active leaf nodes for thespatial index. As part of step 326, the processor can also deactivateleaf nodes that would now be considered to hold “stale” data (ifapplicable). For example, as noted above, it may be desirable to tagleaf nodes that hold “stale” ladar return data with an inactive flag orotherwise purge the leaf node of data so that it will no longerinfluence step 320. In doing so, the system prunes the search process byeffectively subtracting such a stale leaf node from the search process(although the leaf node itself may stay as part of the tree structure200).

After step 326, the process flow would then return to step 300 to repeatitself with respect to the next shot on the shot list. It should benoted that for this repeat pass, if (continuing with the referenceexample from FIGS. 2B-2D) the next ladar pulse shot was targeted towardrange point ACD (and where R is still 3), this next iteration of theFIG. 3A process flow would also use the ladar pulse return data storedin the leaf node for AAD when deciding on the appropriate amount of shotenergy to use when lasing range point ACD. In this fashion, the ladarsystem 100 can adaptively control its shot energy on a shot-by-shotbasis while continuing to take into consideration the prior ladar returndata of nearby pixels in the coordinate space. However, such adaptationneed not necessarily be performed on a shot-by-shot basis if desired bya practitioner. For example, one might adapt pulse energy shot-to-shotif the system discovers that an object is known to be in a certain area(from either prior ladar frames, or a video camera, or a map), or adaptnot all until and if an object is detected, or a blending of the above,with different regions being adapted (or not) based on analysis ofreceived pulse returns.

FIG. 4 shows an example process flow that describes how shot energy fora new ladar pulse shot can be computed in relation to step 320 of FIG.3A. In the example of FIG. 4, it will be assumed that only the returnintensities for prior ladar returns is used to influence shot energyselection; but other information can be included as well if desired by apractitioner. For example, one might include information about thedynamic scene evolution based on ladar vehicle motion. Here an energymanager can explicitly track whether the azimuth and elevation ofprevious shots corresponds to aspects of the current scene that is stillwithin the ladar system's field of view. The closer the prior returnsare located to the edge of the scene, as it is found moving towards therear of the vehicle, the shorter the period of time that one would thenretain that data. One might also include information about the rate ofchange of the scene around the ladar return pixel. This rate of changecould be used to determine how quickly to update the shot energy. Forexample, straight ahead the scene changes very slowly, whereas at theedges the scene changes much faster. Instead of doing a nearest neighborsearch at x,y one could then do a nearest neighbor search at say (x−v_xdt), (y−v_y dt) in order to adapt based on the scene that has moved tothis location vs the scene that was at this location. This is one ofmany examples of how the embodiment of spatial indexes for ladartransmission control can be configured. In addition to motion of thevehicle equipped with ladar, atmospheric attenuation variation can alsobe a factor.

In the example of FIG. 4, steps 400-404 can be pre-processing steps thatare performed prior to interrogation of the subject range point by theladar pulse shot for which the shot energy is to be computed; and steps406-412 are steps that can be performed “live” by the processor as partof steps 320 et seq. from FIG. 3A.

For purposes of explanation, and continuing with the example of FIGS.2B-2D, we will assume the intensity of the 3 leaf nodes that are activeand within R of the subject range point are 0.1, 0.4, and 0.2 (in termsof arbitrary units for purposes of illustration). It should beunderstood that the system can also scale these intensitiesdeterministically by range if desired. Range scaling can be useful if(1) the ladar image is expected to extend over a large range extent and(2) limited interpolation samples are available, such as at thebeginning of a frame and/or after quad tree reset, because range scalingwill help remove/reduce the signal attenuation artifact effects atlonger ranges. A practitioner may also augment the procedure bymodifying the scan radius to include a time lag between each active leafand the anticipated shot time. Also, other modes of active leafmeasurement could be used. For example, if the system hassuper-resolution enhanced steradian estimates, this can be applied tothe interpolation. In the following we assume that the beam spacing, inboth azimuth and elevation, denoted by θ satisfies θ=1 for notationalexpedience, and we assume super-resolved azimuth with respective entries1, 2, 2.5, and that the range to the new voxel is double that ofprevious voxels, all assumed equal in range (or range normalized asdiscussed in above parenthesis), with super-resolved azimuth of 0.1 forAAD.

At step 400, a processor selects a given frame, which may take the formof a defined frame (e.g., a foveation frame) as described in theabove-referenced and incorporated patent applications. The processor canalso select a value for the search radius R at step 400. Once the framehas been selected, the system can determine the ladar pulse shots thatare planned to be taken for the various range points within the selectedframe.

At step 402, the processor finds all of the active leaf node neighborsthat are within R of the targeted range point location for each shot ofthe selected frame. The count of these active leaf node neighbors, whichis a function of returns from lidar shots fired, can be denoted asp. Letus denote by p_({max}) the largest set of possible training samplesavailable. We can find the value of p_({max}) in advance, by simplycounting how many of the range points in the shot list are within adistance R. This is a maximum for p, clearly, since training sets aredefined by active leafs, and active leafs can only appear when there isa shot list associated with said nodes. We note that when the trainingset is sparse, that is few possible leaf nodes are available within afixed distance R, a speedup is available in exchange for memory (and wewill revisit this speedup opportunity shortly).

The maximum degrees of freedom (DOFs) is 6. The degrees of freedom inthis context defines the number of variables in the model. Since we areproposing here a second order polynomial these terms are, consecutively:{constant, Az, El, {Az×El}, Az², El²}. In other words we have theintensity vs shot energy model:

shot(energy)=c ₀ +c ₁ *Az+c ₂ *El+c ₃ Az*El+c ₄ *Az ² +c ₅ *E ²  (1)

The idea is we use past data to find the c₀ . . . c₅ terms and then forany new position we plug in Azimuth (Az) and Elevation (El) and we getthe shot energy. The constant is the “floor” of the intensity. For aroad surface, for example, we would expect very little azimuth (Az) andelevation (El) dependence so the constant term would be large and theother terms small. If we are looking toward the ground we would expectazimuth to change quickly (at least at the road boundary) so that the c₁term is big.

This use of the model in (1) with training as outlined below effectivelyprovides a proportional feedback loop, where the input is scaled basedon the previous measurement (Direct Markov process) or measurements(Markovian State space process). However, instead of simply feeding backto the same pixel (which may not be fired again for some time), theinput (training set) is fed to all neighboring pixels about to fire(which may not have been fired for some time). Due to the multiplicityof neighbors feeding back to any pixel (and spatial variance), we caninterpolate the inputs (the optimal shot energies), regardless of if orwhen these pixels fired. In this manner we have an asynchronousquasi-Markovian, or discrete event system model, which is well-suited toa dynamic, agile, ladar system.

At step 402, the processor can choose the form of the interpolationmodel so that the DOFs are less than or equal to the number of potentialtraining leafs, p, bounded by p_({max}). So, for example, if the valueof p is 6 or greater, the system can use the full set of the parameterlist above; but if p<6, the system will choose a subset of the aboveparameter list so there are not more parameters than samples. Forexample if p=1, we only have one model parameter, and p_({max}) possibleways in which this single return/active-leaf can arise, and for p=2 wehave 2 parameters and (₂ ^(p)) potential active leafs, etc.

In total there will be

${\sum_{i = 1}^{p_{\{\max\}}}\begin{pmatrix}p \\i\end{pmatrix}} = {2^{p_{\{\max\}}} - 1}$

candidate models. For sufficiently small p_({max}) we can choose thesemodels in advance to reduce downstream computation and latency. This isespecially attractive when we have a background frame, and a pop upregion of interest. In such a case the background pattern serves as atraining set for many shots, allowing us to amortize the precomputationcost across many shots. To give a numerical example, let p_({max})=3 forsome range point Q in the list. This means that there are at most 3training samples within range R of Q. Denote the three training samplepoints as R, S, T. Then the possible training sets are reduced to {R},{S}, {T}, {RS}, {RT}, {ST}, {RST}, a total of 7=2³−1 possibilities. Fortraining sets with a single entry we use a constant model, c₀ so in (1)shot energy ∝ intensity in {R} or {S} or {T} respectively. For twoentries in the training set we pick a constant plus azimuth model,yielding c₁, in which case we fit a line to {RS}, {RT}, {ST}, and for{RST} a quadratic. Denote by θ_(apply) the location of the range pointin azimuth for the upcoming shot, and in the two entry case the trainingentries as {a,b} and the training positions in azimuth as θ_(a), θ_(b),then we have:

${{{Shot}\mspace{14mu} {{energy}\left( \theta_{apply} \right)}} \propto {c_{0} + {c_{1}\theta_{apply}}}},{c_{0} = \frac{a + b}{2}},{c_{1} = {\frac{{a*\theta_{a}} + {b*\theta_{b}}}{\theta_{a} + \theta_{b}}.}}$

For the constant and linear examples thus far there is a savings ofthree divisions and three multiplies using pre-indexing with the treestructure 200, in the forming of the c₁θ_(apply) term but behold: Forthe three sample case we have training entries {a, b, c} and we canproceed (using Vandermonde matrix properties) as follows, Set:

${energy} = {{c_{0} + {c_{1}\theta_{apply}} + {c_{2}\theta_{apply}^{2}\mspace{14mu} {which}\mspace{14mu} {{simplifies}\mspace{14mu}\left\lbrack {{computationally}\mspace{14mu} {albeit}\mspace{14mu} {not}\mspace{14mu} {symbolically}} \right\rbrack}\mspace{14mu} {to}\mspace{14mu} {energy}}} = {{R\left\lbrack {\frac{{- \left( {\theta_{b} + \theta_{c}} \right)}\theta_{apply}}{\left( {\theta_{a} - \theta_{b}} \right)\left( {\theta_{a} - \theta_{c}} \right)} + \frac{\theta_{b}\theta_{c}R}{\left( {\theta_{a} - \theta_{b}} \right)\left( {\theta_{a} - \theta_{c}} \right)} + \frac{\theta_{apply}^{2}}{\left( {\theta_{a} - \theta_{b}} \right)\left( {\theta_{a} - \theta_{c}} \right)}} \right\rbrack} + {S\left\lbrack {\frac{\left( {\theta_{a} + \theta_{c}} \right)\theta_{apply}}{\left( {\theta_{a} - \theta_{b}} \right)\left( {\theta_{b} - \theta_{c}} \right)} - \frac{\theta_{a}\theta_{c}}{\left( {\theta_{a} - \theta_{b}} \right)\left( {\theta_{b} - \theta_{c}} \right)} - \frac{\theta_{apply}^{2}}{\left( {\theta_{a} - \theta_{b}} \right)\left( {\theta_{b} - \theta_{c}} \right)}} \right\rbrack} + {T\left\lbrack {\frac{\theta_{a}\theta_{b}}{\left( {\theta_{a} - \theta_{c}} \right)\left( {\theta_{b} - \theta_{c}} \right)} - \frac{\left( {\theta_{a} + \theta_{b}} \right)\theta_{apply}}{\left( {\theta_{a} - \theta_{c}} \right)\left( {\theta_{b} - \theta_{c}} \right)} + \frac{\theta_{apply}^{2}}{\left( {\theta_{b} - \theta_{c}} \right)\left( {\theta_{a} - \theta_{c}} \right)}} \right\rbrack}}}$

Notice that all the terms in brackets can be precomputed in advance. Theenergy evaluation now only takes three multiples, versus about 16multiplies/divides. In terms of multiplies and divides the net savingswith precomputing is then 11 versus 22 multiplies if done directly.

To take a specific case of how the above processing might be employed,let us assume we choose to reset the quad tree at each frame, and wehave a fixed distance R that we scan for neighbors to build our setpoint. At the beginning of a frame there will be no prior shots, so wemust use an initial condition for the first shot (or update from thelast frame before resetting to zero the leaf contents). Assuming that wescan in a roughly left-right up-down fashion we will see that after afew initial conditions we have enough neighboring cells to obtaininterpolation from the current frame. To begin we will have a low modelorder, starting with just a constant, then adding azimuth, thenelevation, then the cross term, then finally quadratic terms. Byprecomputing the Cholesky factor we can keep the interpolation latencylow enough to have updates within a frame. Of course this works best ifthe range point list is configured so that there tend to be numerouspoints within a region of radius R, and that these points are collectedquasi-monotonically in time. This will be the case for a raster scan,foviated patterns, and most uniform random patterns.

For each model, the processor sets up and solves the regression problemthe coefficients in equation (1) above. We denote the right hand siderow vector by b^(t) and the left hand row vector by c where t denotestranspose. The solution for the regression equation A c≈b in Euclideannorm is:

c=(AA ^(t))⁻¹ Ab.  (2)

Here b is the vector of prior data c is the vector of coefficients in(2), and A is the model matrix. (The practitioner will notice A would beVandermonde if the sole cross term Az*El vanishes.)

For the case of (2) with only one parameter other than the constant term(which we denote by θ₀) we can express this, for a random,representative choice of data b as:

$\begin{matrix}{{{\begin{bmatrix}c_{0} & \theta_{0}\end{bmatrix}A} \approx b},{b = \begin{bmatrix}{.1} & {.4} & {.2}\end{bmatrix}},{A = {{\begin{bmatrix}1 & 1 & 1 \\1 & 2 & \frac{5}{2}\end{bmatrix}c_{0}} = {.05}}},{\theta_{0} = {.1}}} & (3)\end{matrix}$

Note that the matrix A is independent of the intensity measured on priorreturns and depends only on spatial structure of range points. This isthe feature that allows us to precompute this factor, as demonstrated inthe “toy” R, S, T example discussed above.

At step 404, the processor finds the Cholesky factor of theinterpolation map for each of the 2^(p) models. For example, at step404, the processor finds the Cholesky factor of AA^(T):

$L = \begin{bmatrix}1.7321 & 0 \\3.1754 & 1.08\end{bmatrix}$

The system has now exhausted all that it can do a priori. Once the ladarsystem begins taking shots, two new pieces of the puzzle will emerge.The first is that the system will know if a pulse reflection is detectedon each shot. If yes, then k, the number of model nodes that are active,which is initially set to zero, will be updated, for each of thepossible node to which the shot is a child. Second, the system will getan intensity return if a pulse reflection is detected. The range pointswith returns for these p shots defines which of the 2^(p) nodes areactive and therefore which model to use for interpolation.

The system can now begin to execute the shot list. At step 406, theprocessor (at time T) interpolates the desired shot energy usingintensity data from the k active leaf node neighbors and thepre-computed Cholesky factors. With our running example from above, step406 can compute the a posteriori optimal shot energy as:

energy=(0.05+0.1*0.1)/2=0.035

In terms of the overall system parameters we have:

A posteriori optimal energy=shot energy out*intensity set point/returnintensity.

This energy allotment scheme can be seen to be valid/efficacious inpractice as follows: when the anticipated return intensity is low weneed more energy. The intensity set point is the dimensionless scalingfor the energy output from prior collections which adjusts for returnintensity. This is akin to a traditional proportional feedback loop,where the input is scaled based on the previous measurement. However,instead of simply feeding back to the same pixel (which may not be firedagain for some time), this input is fed to all neighboring pixels aboutto fire (which may not have been fired for some time). Due to themultiplicity of neighbors feeding back to any pixel (and spatialvariance), we interpolate the inputs (the optimal shot energies).

The computational expediency of this solution is predicated by therelatively few comparisons in FIG. 2A. Suppose we have a radius suchthat the maximum depth we need to search is R. Next assume that a givenactive leaf node is uniform across this space. Let r=log₄(R). UsingBernoulli statistics, and Bayes' law, we find that the average searchcount for p active leaf/voxels is found to be:

${{p{\sum_{{i = 0},\ldots \mspace{14mu},r}\frac{1}{2^{r}}}} \approx r} = {p\; {{\log_{4}(R)}.}}$

The linear equations in equation (3) are solvable using, the Choleskyscheme, of computational complexity

$\frac{p^{3}}{6}.$

Using the fact that A is independent of the ladar returns, we canprecompute A, and indeed (A^(T)A)⁻¹A^(T), for every scheduled shot inthe frame, for every model order. We can further reduce complexity byexpressing (AA^(t)) as LL^(t) where L is the lower triangular Choleskyfactor. This allows us to solve equations involving (AA^(t))⁻¹=L^(−t)L⁻¹by double back solving at a cost of only p² operations. If DOF andintensity samples are equal in quantity, then (AA^(t))⁻¹ A^(T) is squareand can be expressed as a lower times an upper triangular matrix againstresulting in p²⁰ perations. This speed up allows for microsecond-scaleupdate rates, in firmware or hardware implementations. For example, ofthe speedup, one might have a 3 usec update rate ladar system (˜330,000KHz shot rate), with a 300 m range extent, leading to 2 usec for thelast pulse return, giving only 1 usec before the next shot is taken onaverage. With 6 shots used for interpolation of a six DOF model, thecomplexity is about 36 operations, which can be easily met with a 100Mflop device. With a quad tree index, the firmware or hardware is neededmore for data transfer than raw operation counts. Without the quad treeindex, a single sort across a 1M voxel addressable point cloud wouldinvolve 1,000 comparisons per 3 us, thereby stressing the limits of mostportable low cost general purpose computers, and this is without thefrequently needed further complexity of retaining data over multipleframes. With a quad tree index, and R=16, each search now requires only16 comparisons per 3 us.

At step 408, the ladar system transmits the ladar pulse shot toward thetargeted range point (e.g., AAD) with the shot energy computed at step406. If the ladar system detects a return from this ladar pulse shot,the ladar system updates the value for k and also updates theinterpolation model (step 410). Specifically the size of c grows and Ahas an added column. As is the case whenever a return is obtained, thetree is updated as well. So long as there are additional shots in theshot list for the frame, the process flow can return to step 406 andrepeat for the next shot on the shot list. However, once all of theshots for the subject frame have been completed (step 412), the processflow can return to step 400 to select the next frame.

While FIG. 4 describes an example process flow for computing the shotenergy based on an interpolation of the intensities of nearby priorladar return intensities, it should be understood that other techniquescould be used. As discussed previously, the choice of technique involvesthe trade of precomputation in exchange for less recurrent cost. Whetherone favors the former or the latter depends on how fast one expects thetree to be updated, or reset/reinitiated, as well as how sparse weexpect it to be. A technique that minimizes precomputation is to ratherbuild the data matrix A from scratch, and solve equation (3) fromscratch every time the shot energy is updated. This has the advantagethat no Cholesky factor needs to be stored or updated, but recurringcosts are now higher.

Furthermore, as noted above, the spatial index can be expanded todimensions greater than 2 if desired by a practitioner, in which caseadditional search features such as range could be included in thedecision-making about which range points are deemed “nearby” a subjectrange point. Likewise, an extension from a quadratic form ofinterpolation to a multinomial (higher dimensional) interpolation canalso be employed if desired.

Adaptive Interference Mitigation Using Spatial Index of Prior LadarReturn Data:

Furthermore, the ladar system can adaptively control aspects of itsoperations other than shot energy based on the spatial index of priorladar return data. For example, it can adaptively mitigate various formsof interference using the spatial index. Examples of interferencemitigation can include adaptively adjusting control settings of theladar receiver 104 (e.g., controlling receiver sensitivity viathresholds and the like) and detecting/avoiding occlusions or othernon-targeting conditions via adaptive shot selection. A reason to adaptbased on interference is that the laser can vary in efficiency withtemperature. Furthermore, fiber lasers have hysteresis, meaning that theefficiency, specifically maximum pulse energy available, depends on shotdemands in the recent past.

As noted above, receiver noise can be adaptively mitigated using thespatial index. Examples of ladar receiver control settings that apractitioner may choose to adaptively control using the techniquesdescribed herein may include time-to-digital conversion (TDC) thresholdsand spacings (if the ladar receiver 104 employs TDC to obtain rangeinformation from ladar pulse returns) and analog-to-digital conversion(ADC) thresholds (if the ladar receiver 104 employs ADC to obtain rangeinformation from ladar pulse returns). Examples of a suitable ladarreceiver 104 are described in the above-referenced and incorporatedpatent applications. To process a signal that represents incident lighton a photodetector array in the ladar receiver 104, receiver circuitrywill attempt to detect the presence of ladar pulse reflections in thereceived signal. To support these operations, ADC and/or TDC circuitrycan be employed. A tunable parameter for such a ladar receiver are thethresholds used by such TDC and/or ADC circuitry to detect the presenceofladar pulse reflections. In example embodiments, the ladar system canadaptively control these thresholds based on the spatial index of priorladar return data so that the ladar receiver 104 can more effectivelydetect new ladar pulse returns.

Additional potential sources of interference are other fielded andenvironmentally present ladar systems. The spatial index can be used todetect such interference (which will tend to impact multiple beams dueto Lambertian scattering of the other-vehicle-borne ladar). Identifyingthe presence of such other ladar systems, and using interpolation asdiscussed above to locate such interference can assist in interferencemitigation. In addition to the fact that one can desensitize the ladarsystem through threshold changes, interference can be reduced byshifting the transmit beam direction to retain target illumination butreduce exposure to interference. Such an approach is particularlyeffective when the shift is sub-diffraction limited, i.e. a micro-offset(see FIG. 10B discussed below). This can further be enhanced throughinterferometric enhanced processing. Note we discuss here ladarinterference, but passive clutter interference can also be mitigatedwith the above techniques.

FIG. 5A shows an example process flow where the ladar system adaptivelycontrols the sensitivity or other control settings of the ladar receiver104 using the spatial index of prior ladar return data. In the FIG. 5Aprocess flow, steps 300-318 can proceed as described in connection withFIG. 3A.

After all of the nearby active leaf nodes have been identified andaccessed after step 318, the FIG. 5A process flow can proceed to step520. At step 520, the processor performs interpolation on the returndata in the active nearby leaf nodes to compute a parameter value forthe ladar receiver 104. For example, when adaptively controlling an ADCthreshold, the spatial index can store statistically-aggregated noiselevels (such as average, e.g., root mean squared, RMS noise levels) inactive leaf nodes, and set the new shot detection threshold for theladar receiver 104 to be a fixed multiple of this value, based oninterpolation of radial proximate voxel returns. If desired, apractitioner can further expand this to create adaptive range-dependentthresholds, using octagon trees (octrees for short) to encode range if ahighly granular dependency is desired. As an example, it may desirablefor the system to break the voxel returns into several range segments(e.g., 100 range segments) extending from a fourfold quad tree indexwhich encodes azimuth and elevation, to an eight-fold octree index whichencodes azimuth, elevation and range. Similar interpolation techniquescan be used to compute an adaptive TDC threshold, although it isexpected that RMS noise levels would not be available a priori and mustthusly be computed on the fly, by numerically averaging buffered TDCs(such as available from Texas Instruments), adding to recurrentcomputational costs and TDC hardware costs.

At step 522, the processor adjusts the relevant control settings for theladar receiver 104 based on the parameter value computed at step 520.For example, if a register in the ladar receiver 104 holds a thresholdvalue to use for comparison operations by the TDC and/or ADC circuitry,the processor can write the adaptively computed threshold value fromstep 520 into this register. In doing so, the ladar receiver 104 is thenready to receive and process a new ladar shot return.

At step 524, the ladar transmitter 102 transmits a new ladar pulse shottoward the targeted range point. At step 526, the ladar pulse reflectionfrom the targeted range point is received by the ladar receiver 104. Theladar receiver 104 then processes the received reflection using theadjustably controlled parameter value from step 522 to obtain ladarreturn data for the current ladar pulse shot. As noted above, this caninvolve the ladar receiver using a threshold value from step 522 that isstored in a register of the ladar receiver 104 to influence theoperation of TDC and/or ADC circuitry.

Thereafter, as noted above with reference to step 316 of FIG. 3A, theprocessor updates the spatial index with the newly obtained ladar returndata (and it may also deactivate active leaf nodes from the spatialindex as might be appropriate in view of any rules being used tomaintain fresh ladar return data in the spatial index). The process flowcan then return to step 300 for the next shot on the shot list.

The spatial index can be used to control any aspect of the receiver thatmay be spatially dependent. This can include maximum depth of range(once we hit the ground no pulses behind (under) the ground can beobtained, maximum signal return (which can be used to adjust dynamicrange in the digital receiver), atmospheric attenuation (rain and fogcan be angle dependent).

FIG. 5B shows an example process flow where the steps of collectingtraining data above dashed line 350 employ an alternate reverse searchof the spatial index (from leaf node-to-root node). Such an alternatetechnique for searching the spatial index can be useful if the processflow only needs to find a fixed number of training samples. For anexample implementation, to find nearest neighbors (NB for short), theprocessor can perform a top down search, from the new range point. Thesystem can use a stack to keep track of the nodes that it wishes toexplore. The processor can traverse only active branch nodes (which inturn lead to active leaf nodes) that also index a space within distanceR from x, y. A node that satisfies both these conditions is pushed ontothe stack. When a node is popped off the stack for consideration, theprocess repeats, checking all its children nodes for the aboveconditions; if the children are instead leaf nodes and pass the check,the processor has found a NB and can store that leafs index. Bytraversing the tree with this procedure, the processor finds NB leafnodes of the targeted range point that are within the distance R of thetargeted range point (which may include x, y if it is active), withoutexhaustive search, thereby saving computations over the tree searchingtechniques of FIGS. 3 and 5A.

For illustrative purposes for an example run of FIG. 5B, we set a fixednumber of needed training samples, Ntrain, to Ntrain=2, and we set thevalue R=4. As in FIG. 3A, we begin with steps 300 and 302. Step 304 inFIG. 3A is now replaced in FIG. 5B with step 502, whereby we start thesearch at the new range point, and again as in FIG. 3A we initialize thetraining list to the empty set, [ ]. We assume for our example that thequad tree index has already been constructed, as in FIG. 2D, withoutAAD, the new target cell yet added. Next step 306 in FIG. 3A is replacedby 504, whereby we update the training list with all siblings of the newtraining cell that are within R of said cell. In our case, per FIGS.2B-2D, step 504 results in ADA being added to the training set, as wellas ADD. Since Ntrain=2 in this simple example, the process flow can beended, thus using only 4 multiplies for the search and only accessingnode A; whereas in FIG. 3A the process flow needed to access all ofnodes A, B, C, D and further sub-branches. If N train were larger than2, we would need to move to step 506, to inspect siblings of the parentnode of the targeted pixel, to test if they lie within the distance R.One continues thusly, down the stack, 508, until the list size equalsthe required value Ntrain, stage 510.

Accordingly, it should be understood that multiple techniques can beused for searching the spatial index to find nearby prior ladar returndata. Furthermore, while the reverse search technique is described inFIG. 5B with reference to adaptive control of receiver parameters; itshould be understood that the reverse search technique could also beapplied to other forms of adaptive control (such as adaptive shot energycontrol with reference to steps 320 et seq., etc.).

FIG. 6 shows an example process flow where the ladar system adaptivelyadjusts its shot selection using the spatial index of prior ladar returndata. Any of a number of reasons could arise where it may be desirablefor the ladar system to depart from its planned, ordered targeting ofladar pulse shots. For example, there may be grime, dirt, mud, droplets,ice, insect debris, or other occlusions on the outside lens/scope of theladar system such that the quality of signal reception over certainspatial regions of the ladar system's field of view is degraded. In suchsituations, it may be desirable to adjust the ladar pulse shot selectionto “shoot around” the grime/dirt/occlusion. Thus, if there is an oilydirt speck on the outside lens that degrades the ability of the ladarsystem to send/receive with respect to pixels at locations CBA and CBC(see FIG. 2B), it may be desirable for the ladar system to replace aplanned ladar pulse shot targeting location CBA with a ladar pulse shotthat targets ADC (or some other location near CBA). Additional sourcesof noise or signal error could be faulty photodetector cells or lenscracks that would cause a particular region of the observed field ofview to be obscured. As an example, such localized obscurants can bedetected by identifying, for example, disproportionately low intensityreturns that are relatively time-invariant, but persistent in azimuthand elevation. Although it should be understood that if the lens hasmaterial on it that reflects light back into the receiver, then a higherintensity might indicate obscurants.

Similarly, if the ladar system is able to learn that there are objectsin the field of view that may become damaged if illuminated with ladarpulses (as determined from the prior ladar return data), the ladarsystem can adaptively adjust its shot schedule to shoot around suchobjects. Thus, if there is a nearby smart phone with a camera that couldbe damaged if irradiated with a ladar pulse shot, the ladar system candetect the presence of such a smart phone in the prior ladar return datausing image classification techniques, and then make a decision toadaptively adjust the ladar pulse shot schedule to avoid irradiating thecamera's detected location if necessary.

In another example where the ladar system is deployed in an autonomousvehicle used in outside environments (such as an automobile), it may bedesirable to include wiper blades or the like to periodically clean theoutside surface of the ladar system's optical system. In such instances,it is desirable to not illuminate the spatial regions in the field ofview that would be blocked by the wiper blades when the wiper blades arein use. Thus, if the ladar system hits the wiper blade with a ladarpulse shot, the return data from that shot is expected to exhibit alarge intensity magnitude (due to limited beam spreading and thereforelimited energy dissipation) and extremely short range. Thus, the priorreturn data in the spatial index can be processed to detect whether awiper blade is present in a particular region of the field of view, andthe ladar system can adjust the shot schedule to avoid hitting the wiperblades, while minimizing beam offset. More perception/cognition-basedadaptive shot selection schemes can be envisioned. For example, in heavyrain, the video can detect spray from tires, or wheel well spoilers onlarge trucks, and schedule shots so as to minimize spray from saidobjects. Note that techniques for shot timing for the avoidance of wiperblade retro direction can also be used to avoid retro directive returnsfrom debris on the first surface of the transmit lens. Furthermore, theshot scheduler can be adjusted to fire immediately after the blade hascrossed an area to be interrogated, since the lens is cleanest at theexact moment after the wiper has passed.

In the FIG. 6 process flow, steps 300-318 can proceed as described inconnection with FIG. 3A. After all of the nearby active leaf nodes havebeen identified and accessed after step 318, the FIG. 6 process flow canproceed to step 620. At step 620, the processor performs analyzes thereturn data in the active nearby leaf nodes to determine whether anocclusion or other non-targeting condition is present in that spatialregion (see the discussion above regarding examples instances where itmay be desirable to adjust the shot schedule). An example of occlusiondetection is to explore optical flow as the ladar-equipped car movesthrough a scene. Without occlusion objects detected at point A on aprevious shot can be expected to appear at point B at a new shot, basedon relative motion between said object and the ladar. If this seconddetect fails to appear over multiple trials, then an occlusion in likelyto be the most plausible culprit.

If step 620 results in a conclusion by the processor that an occlusionor other non-targeting condition is present, the process flow returns tostep 300 and a new ladar pulse shot is selected from the shot list. Aspart of this, the ladar shot that was avoided because of this branchingmay be returned to a later spot on the shot list. However, apractitioner may choose to employ other adjustments to the shotschedule. For example, the avoided ladar pulse shot could be droppedaltogether. As another example, the avoided ladar pulse shot could bereplaced with a different ladar pulse shot that targets a nearbylocation. If step 620 results in a conclusion by the processor that anocclusion or other non-targeting condition is not present, then theprocess flow proceeds to step 622.

At step 622, the ladar transmitter 102 transmits a new ladar pulse shottoward the targeted range point. At step 624, the ladar pulse reflectionfrom the targeted range point is received by the ladar receiver 104. Theladar receiver 104 then processes the received reflection to obtainladar return data for the current ladar pulse shot. If desired by apractitioner, the ladar receiver 104 could also be adaptively controlledbased on the content of the spatial index as discussed in connectionwith FIGS. 5A and 5B. Thereafter, as noted above with reference to step326 of FIG. 3A, the processor updates the spatial index with the newlyobtained ladar return data (and it may also subtract active leaf nodesfrom the spatial index as might be appropriate in view of any rulesbeing used to maintain fresh ladar return data in the spatial index).The process flow can then return to step 300 for the next shot on theshot list.

Furthermore, while FIG. 6 shows the tree searching technique of FIG. 3Abeing used to collect the training samples; it should be understood thatother tree searching techniques could be used prior to step 620, such asthe one described in FIG. 5B.

FIG. 7 shows an example ladar system 100 that is capable of providingmultiple types of adaptive control based on the spatial index 160.Spatial index 160 can store prior ladar return data in a tree structuresuch as a quad tree index as discussed above. Driving software executedby a processor in the control system can generate requests 752 forranging set of coordinates in the field of view. For example, theserequests can be part of a shot list as discussed above and as describedin the above-referenced and incorporated patent applications. Theprocessor can then execute energy assignment logic 704 using thetechniques discussed above in relation to FIG. 3A to determine how muchenergy to include with a given ladar pulse shot, where thisdetermination is based on an analysis of nearby ladar return data forthe subject shot as found in the spatial index 160. The result can bethe shot requests being augments with a laser charging time for eachshot so that each shot can include a desired amount of energy. The shotscheduling logic 706 can request (756) to fire the laser at the targetedrange point when the scanning mirrors target the subject range point andthe laser is sufficiently charged. If desired, the shot scheduler canalso leverage the spatial index 160 as discussed above in connectionwith FIG. 6 to determine whether any occlusions or other non-targetingconditions are present such that a given ladar shot should berescheduled or dropped. For example, the shot scheduler 706 can bedesigned to accept vector interrupts based on information learned fromthe spatial index 160, where the vector interrupts would change theorder or listing of shots that the scheduler 706 schedules.

Scanning block 708 can include the scanning mirrors and will direct thefired laser to the targeted range point. The scanning block 708 balancestwo conditions for pulse firing. It chooses the time to fire when themirrors are in the correct azimuth and elevation position to hit thetargeted range point. It also ensures, through information it collectsfrom the spatial index 160, that the shot energy level is correct.However, these two criteria can be in conflict, since shot energy varieswith the time between shots. The process of sorting out these criteriacan be controlled by a shot scheduler (which can be referred to as amangler). Since all shot energy levels impact the mangler, tightcommunication between the scan mirrors and the tree structure isdesirable as shown by the lines connecting the scanning block 708 andthe spatial index 160. The ladar pulse 758 will then propagate in theenvironment toward the targeted range point. A reflection 760 of thatladar pulse 758 can then be received by the receiver block 710 via lens712. A photodetector array and associated circuitry in the receiverblock 710 will then convert the reflection 760 into a signal 762 that isrepresentative of the ladar pulse reflection 760. Signal processingcircuitry 714 then processes signal 762 to extract the ladar pulsereturn data (e.g., range 764, intensity, etc.). This ladar pulse returndata then gets added to the ladar point cloud 716 and spatial index 160.Examples of suitable embodiments for scanning block 708, receiver block710, and signal processing circuitry 714 are disclosed in theabove-referenced and incorporated patent applications.

As shown by FIG. 7, the spatial index 160 can be used by many of thevarious blocks in this system to adaptively improve their operations. Asnoted above, the spatial index 160 can be used by the energy assignmentblock 704 to adaptively control how much energy is assigned to eachladar pulse shot; and the spatial index 160 can also be used toinfluence shot scheduling block 706 as well as scanning block 708.Furthermore, the receiver block 710 can use the spatial index 710 toadaptively control its operations (e.g., amplifier gains for theamplifier circuitry that amplifies the signals detected by thephotodetector array in receiver block 710 can be adaptively controlledbased on the nearby prior ladar return data in the spatial index 160).Also, as noted above, the comparison/detection thresholds in the signalprocessing circuitry 714 can be adaptively controlled using the spatialindex 160. Accordingly, it can be seen that the spatial index 160 ofprior ladar return data can be used by ladar system 100 to adaptivelycontrol any of a number of different types of system operations.Furthermore, while the example of FIG. 7 shows the spatial index 160being used to adaptively control several different types of operations;it should be understood that a practitioner may choose to implement thesystem 100 in a manner so that fewer than all of the processing blocksleverage the spatial index 160 if desired.

Further still, with reference to FIG. 7, the ladar system 100 can alsoleverage the spatial index 160 to adjust settings of a camera (e.g., seecamera 1702 in the example ladar system 1700 of FIG. 17) that is used bythe system to support decision-making about which range points should beincluded on the requests 752 that make up the shot list. As explained inthe above-referenced and incorporated patent applications, the ladarsystem 100 may employ a camera to assess which areas within theenvironmental scene are most salient for interrogation by ladar pulseshots. A practitioner may find it desirable to use the spatial index 160to adaptively control various camera settings such as camera lightsettings. As a simple example, suppose at night (or in generally pooroptical visibility), the ladar system detects a car that has lowreflectivity (which is very likely then to be a black-painted car). Theladar system can then instruct the camera to retain the shutter open forlonger than usual, in hopes of capturing a dim black object thatotherwise is likely to be missed.

Example Use Cases

The left frame of FIG. 8 shows an example ladar return image 800 that isderived from a ladar system where no adaptive feedback control toachieve a more uniform illumination of the field of view is notemployed. Image 800 is a 2D projected image of the intensity map in theladar point cloud produced by a ladar system lacking adaptive feedbackfor a given frame. The right frame of FIG. 8 shows a camera image 802 ofthis same scene. As can be seen in ladar image 800, there are severalvertical stripes that are present, and these stripes are not present inthe environment. Instead, these stripes are due to artifacts in theladar return data that arise, at least in part, due to non-uniformitiesin the shot energy of the transmitted ladar pulses.

By contrast, FIG. 9 shows an intensity map, 3D perspective point cloudwhere the ladar system employs adaptive feedback control of shot energyusing the spatial index 160 and FIG. 3A process flow (left frame) aswell as a 2D image of this intensity map (right frame). As can be seenin FIG. 9, most artifacts are either gone or greatly suppressed; and theintra-object standard deviation (IOSD) has declined from an average of20% to less than 4%. IOSD is a metric for intensity uniformity; itdescribes how much the image intensity changes when the object remainsthe same, in other words it separates variation in the ladar (sensor)from variation in the scene itself.

FIG. 10A shows an example process flow where the ladar system employsadaptive feedback control of shot energy (using the spatial index 160and FIG. 3A process flow) to construct a ladar intensity map of a roadsurface. Because the artifacts (e.g., striping effects as shown by FIG.8 (left frame) can be reduced in such a road surface ladar intensitymap, the system is able to better detect obstacles that may be presentin the road. For example, the ambient intensity level in the ladarintensity map can be classified as corresponding to the road surface;and any contiguous regions of intensity that are lower than this ambientintensity can be detected and classified as a potential obstacle.Accordingly, a pedestrian with dark clothing or a tire tread in the roadmay be detected more reliably because of the reduced presence of noiseartifacts in the ladar intensity map. FIG. 10B shows a camera image(left frame) and a corresponding ladar intensity map (right frame) of acar's wheel well, including tire. FIG. 10B specifically shows theintensity as a function of azimuth and elevation at the tire and wheelwell. The intensity is low in the wheel well and higher at the tire. Inthis example, the range is not sufficiently accurate to assist in thetire versus well classification, so intensity was the best availablefeature. This range-insufficiency is also the case for tire debris on aroadway, but with examples embodiments described herein for adaptiveshot energy management for intensity imaging, the intensity informationin the returns can be used to better classify objects in the field ofview, greatly contributing to improved safety. Accordingly, with theexample of FIG. 10B, by employing adaptive feedback control of shotenergy (using the spatial index 160 and FIG. 3A process flow), the ladarsystem is able to produce an intensity map with sufficient sensitivityto distinguish between dark areas of the camera image corresponding tothe tire and the disproportionately darker tire rim wall.

Thus, with reference to FIGS. 10A and 10B, the spatial index can be usedto normalize the intensities over a region in the field of view, andthis can help reveal weak objects that would normally benon-distinguishable from background. For example, an object that is verydark in a ladar image is also likely to be dark in a camera imagebecause the ladar wavelength is likely to be near the visiblewavelength, which means dark object tend to react the same way in both aladar view and camera view (e.g., black cars will appear dim in bothladar and camera views). Accordingly, in embodiments where the ladarsystem includes a camera, the spatial index can be used by the ladarsystem to smooth intensities over a neighborhood of range points in thefield of view/coordinate space; and the ladar system can then controlone or more camera settings as a result of the smoothed returnintensities (and the objects that are newly revealed because of thesmoothed intensities). For example, the camera can be instructed toincrease exposure time to obtain a better image of the weak/dim object.In a powerful example embodiment relating to automobiles, the adaptiveladar system with camera can be used to better detect oncoming cars atfarther distances. If there is a dark car relatively far away in frontof the ladar system, it may be difficult for a conventional ladar systemto distinguish such a faraway car from a dark road surface and/orclassify the object as a car rather than, say, a tree branch. However,by using the spatial index to intelligently control camera settings soas to increase the effective exposure for producing camera images (e.g.,using longer integration with digital cameras or changing the camerapupil mechanically), the adaptive ladar system can accurately classifythe oncoming car at a farther distance, thereby greatly improvingsafety. Furthermore, since increasing the exposure slows down the cameraframe rate (if the camera's aperture remains open longer, this meansfewer frames per second), the ability of the ladar system tointelligently adapt its decision-making regarding when it is desirablethat the ladar system increase camera exposure provides a good balancebetween speed and accurate detection.

FIG. 11 shows an example use case where the ladar system uses adaptivefeedback based on the spatial index of prior ladar return data to reducelaser interference. At step 1, the ladar system contrasts theinterpolation of return intensity over a neighborhood of nearby rangepoints from the spatial index with the measured return for a range pointin the subject neighborhood. If the contrast between the two is beyond adefined threshold, the system can conclude that laser interference ispresent. At step 2, the system can localize this interference by using ashot cluster around the subject range point while also adjusting theazimuth and elevation of the shot for that range point at a next frameto graze the interference gaze angle while still covering nearbypotential objects. In simplified terms, one is using the fact that thereis overlap between ladar beams to “peak” at the side of an object ofinterest similar to how one squints in sunlight to view an object byremoving light from a bright object, even direct sunlight FIG. 12A showsan example use case for an application of adaptive shot energy controlto reduce the damage risk to cell phones or other digital cameras. Asnoted above, the cameras that are included with cell phones can besusceptive to damage if irradiated with a ladar pulse shot because thesmall and thin photodetector arrays in such cameras may not be able tohandle the energy within a ladar pulse shot (typically due to thethinness of the quantum wells in such cameras). Moreover, this dangerexists for cell phone cameras and other digital cameras even if thelaser light is well within the limit that this safe for human eyes.Accordingly, it is desirable for the ladar system to adapt its shotselection and/or otherwise control its shot energy to avoid dosingnearby digital cameras with too much light. In the context of autonomousvehicles, this risk can be considered in an example use case where theladar-equipped autonomous vehicle is stopped at an intersection or in aparking lot and there are pedestrians standing nearby using their cellphones (perhaps even a curious pedestrian who is taking a photograph ofthe autonomous vehicle with his or her cell phone from close to thevehicle's ladar system). The adaptive ladar system described herein canleverage the spatial index to reduce the risk of damaging such a nearbycell phone.

In step 1 of FIG. 12A, the ladar system monitors the ladar return datafrom the spatial index for objects that are within a defined phonedamage range (e.g., X feet). FIG. 12B shows a set of measurements for acell phone (an iPhone 6 in this example) with respect to a ladar system(an AEye development system (AE90) in this example) that quantifiescamera pixel damage distance versus charge time adjustment for ladarshots. The vertical axis shows distance between camera and ladar systemoutput window before camera damage occurs (for standard ladar energy).The horizontal axis shows the set point values with respect to lasercharge time. The dotted horizontal line at roughly 270 mm on the y-axisshows the measured pixel damage distance (from ladar system outputwindow to phone) when the ladar system does not employ adaptive feedbackcontrol as described above. In this example, the risk of damaging thecamera due to light irradiation of the camera becomes greatly reduced atdistances of more than roughly 270 mm (or roughly 10.5 inches) betweenthe camera and the ladar system. The solid line plot (and dotted linelinear curve fit) shows the reduction in range before the camera isdamaged, versus the set point parameter (which defines the amount oflaser energy transmitted versus intensity received). The set point is aunitless, out of 255, ADC quantization of the received signal, for an 8bit byte/ADC. Accordingly, in this example, it can be seen that anexample smart phone would experience camera damage when a ladar pulsewithout energy management strikes the camera from roughly 10.5 inchesaway. However, if adaptive shot energy control is used so that the shotenergy for the subject ladar pulse shot is reduced (as shown by thex-axis where smaller set point values (charge times) would correspond toless shot energy), it can be seen that the adaptive ladar system wouldrequire the smart phone camera be much closer to the ladar system inorder to experience damage. While FIGS. 12A and 12B describe anapplication of adaptive shot energy management to protecting smart phonecameras, it should be understood that these shot energy managementtechniques can also be extended to other areas such as minimizinginterference for free space optics communications and/or ladar-to-ladarpulse return contamination.

Thus, at step 2, the ladar system can use the object monitoring fromstep 1 to adaptively adjust shot energy in a manner that reduces therisk of damaging a smart phone within the target region. For example, ifstep 1 detects a cell phone in the region around range point ABC, theladar system can select a charge time set point value for the ladarpulse targeting ABC as a function of the detected range (e.g., see FIG.12B) so that camera damage risk is reduced. The ladar pulse shot withits energy controlled as per step 2 can then be fired at the targetedrange point.

Subsequently, at step 3, if the ladar return data indicates that thecamera persists in the field of view, the ladar system can then adjustits shot list to exclude illumination of the camera (see, e.g., FIG. 6).

Synthetic Ladar Frames:

In another example embodiment, interpolation techniques and the like canbe used to fill data gaps in a sparse array of ladar return data. Forexample, as described in the above-referenced and incorporated patentapplications, the ladar system 100 can employ compressive sensing todynamically scan the field of view and intelligently target only a smallsubset of range points in the field of view. In this fashion, the ladarsystem can focus its targeting on the areas of interest in the field ofview that are believed to be the most salient for safety and accurateobject detection while avoiding overly shooting into empty space. Withcompressive sensing, the ladar system will produce sparse arrays ofladar return data for each ladar frame. This permits the ladar system100 to exhibit a much higher frame rate while not losing any meaningfuldetection coverage. However, many downstream applications that processladar data to perform image classification and object tracking may bedesigned with a standardized interface with ladar systems. With suchstandardized interfaces, it may be desirable, for softwareinteroperability for instance, for the ladar frames to exhibit somefixed size or minimum size. Accordingly, to make a dynamic ladar systemthat employs compressive sensing compatible with such downstream imageprocessing applications, a practitioner may want to synthetically fillthe sparse ladar frames produced by the ladar system 100 withinformation derived from the sparse array of ladar return data.

FIG. 13A shows an example process flow for creating synthetic ladarframes in this fashion. At step 1300, the system decides whether tocollect a new ladar frame. If so, the process flow proceeds to step1302. At step 1302, the ladar system 100 uses compressive sensing todynamically target a number of range points as discussed in theabove-referenced and incorporated patent applications. This produces asparse array of ladar return data.

At step 1304, a processor synthetically fills a ladar frame based on thedata in the sparse array. In doing so, the processor is able to fill theempty gaps (the non-targeted range points) in the field of view betweenthe various range points that had been targeted with ladar pulse shots.Interpolation techniques such as those discussed above can be used tofill in these gaps between ladar pulse return data from the sparsearray. The gap filling can be done using any data extrapolation method,examples of which are available in MatLab and other software packages,such as polynomial interpolation, or iterative projection onto convexsets. The spatial index of prior ladar return data can be leveraged atstep 1304 to accelerate the calculations that will drive step 1304. Forexample, if the processor can use the ladar return data from the spatialindex for a plurality of range points that are near a blank range pointto be synthetically filed to facilitate an interpolation or otherfilling computation that allows the processor to synthetically computehypothetical synthetic return data for the non-targeted range point.This process can be iteratively repeated for different non-targetedrange points (or non-targeted regions in the coordinate space), untilthe ladar frame is deemed full (step 1306). Once full, thesynthetically-filled ladar frame can be output for consumption by adownstream image processing application such as an image classificationor object detection application.

It should be noted that because step 1304 does not require anyadditional scanning by the ladar system's mirrors as well astransmit/receive propagation delays, it is expected that step 1304 canbe performed with low latency relative to the time that would be neededto physically scan and interrogate the non-targeted range points vialadar pulses, particularly if sufficiently powerful compute resourcesare used such as parallel processing logic. In this fashion, it isexpected that the frame rate for the FIG. 13A process flow will befaster than the frame rate available from conventional ladar systemsthat employ a conventional fixed raster scan to produce ladar frames.

FIG. 13B shows both image 1050 of a sparse array of ladar return dataand an image 1052 of a synthetically-filled frame derived from thesparse array of ladar return data. Image 1050 in this examplecorresponds to a sparse array that covers around 30% of the field ofview with ladar returns from interrogated range points. Image 1052 showsa synthetic full frame corresponding to the same sparse array that wasconstructed using a quad tree index of the sparse array andinterpolation of the ladar return data in the sparse array. As notedabove, such a synthetic frame may be more compatible with existing imageprocessing/object detection applications, and it is also more intuitivefor human consumption.

With the example approach of FIG. 13A as exemplified by FIG. 13B, thesystem can use super-resolution to interpolate, which allows for thedropping of ladar shots that would otherwise be taken. For example, thesystem can drop randomly selected ladar shots. With reference to FIG.13B, the image at left can be obtained after dropping around ⅔ of theladar shots. With super-resolution, the image at the right of FIG. 13Bcan be obtained, which is virtually indistinguishable from a fullcollection in quality, but requires approximately ⅓ of the time tocollect as compared to the full collection.

In another example embodiment, the synthetic ladar frame can begenerated in coordination with a rolling shutter technique. An exampleof such a rolling shutter technique is shown by FIG. 14. Through therolling shutter technique, the system can quickly find motion while atthe same time creating a sparse data set. As an example, suppose theaddressable shot list for the ladar system has 1000 rows; in other wordsthere are 1,000 different values in elevation (y value using earliernotation) that the laser can point at when firing shots. With a rollingshutter technique, the ladar system will skip rows in the scan patternat regular intervals (see step 1400), and then repeat the process in theother direction (see step 1406), and so on until all of the rows arecaptured. For example, with the 1000 row example, the ladar system canscan the top horizontal line (row 1), then scan the 4^(th) horizontalline (row 4), and so forth until the ladar system scans horizontal lines997 and 1000 (rows 997 and 1000) respectively. Then, the ladar systemwould scan upward, starting from the third to last horizontal line (row998), then horizontal line/row 995, and so on up to the horizontalline/row that is 2^(nd) from the top (row 2). Next, the ladar systemscans downward from the third horizontal line/row (row 3), to theseventh horizontal line/row, 11^(th) horizontal line/row, and so onuntil horizontal line row 999 (at which point the ladar system will havescanned every row. With such an approach, the ladar system can still becreating a sparse image data set if the ladar system is samplingsparsely within the horizontal lines/rows (even though all horizontallines/rows are covered).

With such a “rolling” ladar frame, there is a benefit in that the ladarsystem gets near every point in the image roughly twice as fast as onewould by scanning each row in succession. Therefore, anything thatenters the frame which exhibits a size of three or more horizontallines/rows or more can be detected twice as fast. New ladar frames canbe generated each time the shot list has been scanned in a givendirection (step steps 1402 and 1404). Synthetic filling techniques asdescribed at step 1304 of FIG. 13A can be applied at step 1404 to filleach frame.

Furthermore, if these synthetic frame techniques are combined with fastvelocity estimation as described in U.S. patent application Ser. No.16/106,406, entitled “Low Latency Intra-Frame Motion Estimation Based onClusters of Ladar Pulses”, filed Aug. 21, 2018, the entire disclosure ofwhich is incorporated herein by reference, the ladar system can obtainan interpolated dense image of a static background scene and targetmotion at the same time. Velocity estimations of a moving object in theimage can be obtained using the current position of the object in thecurrent synthetic frame and the prior position of the object in theprevious synthetic frame, assuming that the radius parameter R discussedearlier is less than the width and height of the moving object. Forexample, the current synthetic frame can be based on an ascendingtraversal of the shot rows while the prior synthetic frame can be basedon the immediately preceding descending traversal of the shot rows.Assuming that the moving object encompasses three or more horizontallines/rows, this permits the ladar system to estimate velocity based ontheir relative positions in the synthetic frames. Given that thesynthetic frames are being generated faster than a conventional fullladar frame, this improves the speed by which the ladar system candetect and track object velocity. Also, if desired, a practitioner neednot reverse the scan direction for each traversal of the shot list atstep 1406.

Optical Flow for Target Tracking Using Spatial Index Interpolation:

In another example embodiment, the spatial index-based interpolationtechniques described herein can be used in combination with optical flowtechniques for target tracking. FIG. 15 shows an example of optical flowas it can relate to a moving vehicle 1520. In the example of FIG. 15,the scene from the perspective of a moving vehicle equipped with a ladarsystem is shown. The scene ahead is shown in the azimuth (cross range)and elevation (height) dimensions. Arrows 1502 and 1504 show how a pixel1500 at the road horizon will “flow” or “split” into a left trajectory(see arrow 1502) and a right trajectory (see arrow 1504) as motionforward continues. Techniques for the use of optical flow for machinevision based on images from video cameras are described inMeinhardt-Llopis, et al., “Horn-Schunck Optical Flow with a Multi-ScaleStrategy”, Image Processing On Line, Jul. 19, 2013, 22 pages, the entiredisclosure of which is incorporated herein by reference.

However, the use of optical flow in ladar systems is complicated by atleast two factors: (1) with ladar systems, only a small fraction of thescene is usually illuminated/interrogated (e.g., even with raster scanapproaches, beam steps may be 1 degree with a beam divergence on theorder of 0.1 degree)—in which case much of the information needed forfull scale optical flow is not available and must be estimated, and (2)unlike video systems (which operate in angle/angle (steradian) space—andwhich delivers angular pixel sizes that are independent of range), ladarsystems work with voxels (volume pixels) that exhibit a size which growswith depth. Because the sizes of voxels grow with depth, the opticalflow calculation for ladar systems will need to account for aligningdata at different distances. For example, with reference to FIG. 15, thesame car 1506 is shown on the same image as it appears on a road at twodistinct points in time. At near range, it can be seen that the beams1510, 1512, and 1514 for different ladar shots can separatelyinterrogate the taillights and license plate. However, at longer range,it can be seen that the beams 1520, 1522, and 1524 will be overlappingwith respect to the taillights and license plate. Accordingly, at longerranges, interpolation or super-resolution is desired because variousobjects in the scene (such as taillights and license plates) are notseparated across beams.

Furthermore, while the depth information that is available from a ladarsystem helps with object tracking, it can be a complicating factor forarresting self-motion of the ladar system. However, the ability to usethe spatial index as discussed above to interpolate returns not onlyhelps solve complicating factor (1) discussed above, but it also allowsfor the system to generate fixed size optical flow voxels, which helpssolve complicating factor (2) discussed above.

The spatial index-based interpolation can generate fixed size voxels asfollows. If the object (car) is at close range, then the shotsthemselves are fine-grained, and no interpolation is needed. At longerrange—where the actual data voxels are spaced wider apart—the spatialindex-based interpolation techniques can be used to obtain afinely-spaced sampling. For example, suppose at 10 m the shot griddelivers samples spaced every foot apart. At 100 m, the same shot listproduces voxels 10 feet apart. But, given this widely-spaced data, onecan interpolate using the spatial index (e.g., see equation (3) above)to effectively resample at one foot spacing.

FIG. 16 shows an example process flow where the spatial index of priorreturn data can be used in combination with optical flows to adaptivelycontrol shot selection. At step 1600, the ladar system fires ladar pulseshots for Frame k, and it forms a point cloud from the return data forFrame k.

At step 1602, the system determines the motion from Frame K to previousframes using knowledge of the ladar system's velocity. Step 1602 can beperformed using techniques such as arrested synthetic aperture imaging(SAI), or image stacking. For example, in principle, when a person is apassenger in a car driving past a wheat field, that person will noticethe wheat ahead appears to move must faster than wheat directly to theperson's side, if in both cases a reference point (e.g., wheat stock) ispicked at the same radial distance. Arrested SAI uses this fact,combined with knowledge of the car motion, to align (stack) images frameto frame. By determining the motion between frames, the system canregister the current frame to previous frames for static objectsdepicted in those frames (e.g., see 1502 and 1504 in FIG. 15 for pixel1500). This allows for optical flows to be defined across frames for thestatic objects that are depicted in the frames. So, for example, ifthere are two images, A and B, of a clump of wheat stored in a spatialindex, each pixel of each image can be tagged with apparent velocity.

Then, at step 1606, the system can determine whether an optical flowline maps the current shot position for the ladar system to a positionin the previous frame for which there is not a ladar return. If the shotposition maps to an optical flow line for which there is not priorreturn data, the system can leverage the spatial index of prior returndata to interpolate the return data for the current shot position usingthe spatial index interpolation techniques described above. There areseveral techniques for choosing what points to use as a basis forinterpolation. One example is to use the points which are the subjectpoints in the prior frame. Another technique is to use edge detection tofind the boundary of an object, and then store the associated voxelswhich form that boundary edge from the prior frame as the basis forinterpolation in the current frame. By doing so for the optical flowlines found at step 1604, the system is able to build a static sceneintensity (clutter) map for the static aspects of the scene.

This leaves the objects in the scene that are moving. For these objects,the ladar system can include a camera that produces camera images of thescene (e.g., camera data such as video RGB data). Moving objects withdefined characteristics can then be detected within the camera images.For example, moving objects with high reflectivity (e.g., the taillightsand license plate shown in FIG. 15) can be detected in the camera data,and by processing camera data over time, the system can predict thefuture positions of these moving objects (e.g., by tracking changes inlocation within motion-reconciled camera images over time). Theprediction can be achieved using knowledge about the ladar system'smotion (e.g., the motion of a vehicle equipped with the ladar system).For example, if the ladar system is known to be moving at 10 m/s, and aroad sign 50 m in front of the ladar system is seen, a prediction can bemade that, barring any acceleration, the sign (assumed stationary) willbe 40 m away in one second in the future. Thus, with reference to FIG.15, if car 1506 is speeding away from the ladar system, the system canpredict that the taillights and license plate which are at 1510, 1512,and 1514 at time t=t0 will be located at 1520, 1522, and 1524 at timet=t0+x.

Next, at step 1610, the system can schedule ladar shots that target thepredicted future positions for those moving objects as determined atstep 1608. Accordingly, with reference to the example of FIG. 15, forFrame k+1, the system can define ladar shots that will target 1520,1522, and 1524 based on the prediction that this will be the location ofthe taillights and license plate at that time. In this fashion, opticalflow techniques can be combined with the spatial index interpolationtechniques to adaptively control shot selection for the ladar system.With this adaptive shot selection, the ladar system can reduce thenumber of ladar shots that are needed for the scene while retaining oraugmenting the shots that are targeted to the important and salientparts of the scene.

While FIGS. 15 and 16 describe an example of optical flow for thehorizon as well as points on a fixed vehicle, it should be understoodthat other embodiments are possible. For example, spatial indexinterpolation for ladar systems can be used to track red objects in avideo camera. For automotive applications, red is almost alwaysassociated with highly reflective materials that provide keyinformation, such as brake lights on a car or stop signs. Accordingly,using this color channel for adaptive ladar sensing can be very useful,and there is little chance that an erroneous red “clutter” sample willarrive and confuse the frame-to-frame tracker.

While the invention has been described above in relation to its exampleembodiments, various modifications may be made thereto that still fallwithin the invention's scope. Such modifications to the invention willbe recognizable upon review of the teachings herein.

1. A ladar system comprising: a ladar transmitter configured to transmita plurality of ladar pulse shots into a coordinate space toward aplurality of range points such that less than a full set of range pointswithin a ladar frame are targeted with the ladar pulse shots; a ladarreceiver configured to receive incident light and detect ladar returnsfrom the ladar pulse shots based on the received incident light; and aprocessor configured to synthetically fill a ladar frame based on thedetected ladar returns such that the ladar frame includes syntheticreturn data for range points in the ladar frame that were not targetedwith ladar pulse shots; and wherein the ladar transmitter is furtherconfigured to employ a rolling scan of the ladar frame to supportgeneration of a plurality of the synthetically-filled ladar frames. 2.The system of claim 1 wherein the ladar transmitter is furtherconfigured to employ compressive sensing within the ladar frame totransmit ladar pulse shots into the coordinate space toward a pluralityof range points such that less than the full set of range points withinthe ladar frame are targeted with the ladar pulse shots.
 3. The systemof claim 1 wherein the processor is further configured to compute thesynthetic return data based on a plurality of interpolations from thedetected ladar returns.
 4. The system of claim 1 wherein the processoris further configured to compute, for each of a plurality ofnon-targeted range points in the ladar frame, the synthetic return datafor that non-targeted range point based on the detected return data fora plurality of range points in a defined vicinity of that non-targetedrange point.
 5. The system of claim 4 further comprising: a memoryconfigured to store data about the detected ladar returns from priorladar pulse shots in a spatial index, the spatial index associating thedetected ladar return data with a plurality of locations in thecoordinate space; and wherein the processor is further configured toretrieve the detected return data in the defined vicinities for use inthe synthetic return data computations from the spatial index based onthe locations of the non-targeted range points in the coordinate space.6. The system of claim 1 wherein the detected ladar return datacomprises intensity data for the targeted range points, and wherein thesynthetic return data comprises synthetic intensity data for thenon-targeted range points.
 7. The system of claim 1 wherein the detectedladar return data comprises range data for the targeted range points,and wherein the synthetic return data comprises synthetic range data forthe non-targeted range points.
 8. The system of claim 1 furthercomprising: an image classification application for execution by aprocessor, the image classification application configured to processthe synthetically-filled ladar frame to classify an object that isdepicted in the synthetically-filled ladar frame.
 9. The system of claim1 wherein the ladar transmitter is further configured to employ therolling scan of the ladar frame using compressive sensing to support thegeneration of synthetically-filled ladar frames.
 10. The system of claim9 wherein the rolling scan skips two lines in a field of view for theladar frame in ascending and descending shot schedules.
 11. The systemof claim 9 wherein the processor is further configure to estimate avelocity of an object depicted in the synthetically-filled ladar framesbased on comparative positions of the object in the synthetically-filledladar frames.
 12. The system of claim 11 wherein a synthetically-filledladar frame corresponding to an ascending scan of a field of view and asynthetically-filled ladar frame corresponding to a descending scan of afield of view are used by the processor for the velocity estimation. 13.The system of claim 11 wherein the object exhibits a size that is atleast as large as a skip length for the rolling scan.
 14. The system ofclaim 1 wherein the ladar transmitter includes a plurality of scanablemirrors; wherein the processor is further configured to dynamicallyschedule a plurality of ladar pulse shots for the ladar transmitter totarget a plurality of range points in the coordinate space; wherein theladar transmitter is further configured to (1) controllably scan thescannable mirrors to target the ladar transmitter at the targeted rangepoints in accordance with the scheduled ladar pulse shots, and (2)transmit a plurality of ladar pulse shots toward the targeted rangepoints via the controllably scanned mirrors.
 15. The system of claim 14wherein the processor is further configured to perform the dynamicscheduling on a shot-by-shot basis.
 16. A method comprising: a ladartransmitter transmitting a plurality of ladar pulse shots into acoordinate space toward a plurality of range points such that less thana full set of range points within a ladar frame are targeted with theladar pulse shots; a ladar receiver receiving incident light; the ladarreceiver detecting ladar returns from the ladar pulse shots based on thereceived incident light; and a processor synthetically filling a ladarframe based on the detected ladar returns such that the ladar frameincludes synthetic return data for range points in the ladar frame thatwere not targeted with ladar pulse shots; and wherein the transmittingstep includes employing a rolling scan of the ladar frame to supportgeneration of a plurality of the synthetically-filled ladar frames. 17.The method of claim 16 wherein the transmitting step comprises the ladartransmitter compressively sensing within the ladar frame by transmittingladar pulse shots into the coordinate space toward a plurality of rangepoints such that less than the full set of range points within the ladarframe are targeted with the ladar pulse shots.
 18. The method of claim16 wherein the synthetically filling step comprises the processorcomputing the synthetic return data based on a plurality ofinterpolations from the detected ladar returns.
 19. The method of claim16 wherein the synthetically filling step comprises the processorcomputing, for each of a plurality of non-targeted range points in theladar frame, the synthetic return data for that non-targeted range pointbased on the detected return data for a plurality of range points in adefined vicinity of that non-targeted range point.
 20. The method ofclaim 19 further comprising: storing data about the detected ladarreturns from prior ladar pulse shots in a spatial index, the spatialindex associating the detected ladar return data with a plurality oflocations in the coordinate space; and the processor retrieving thedetected return data in the defined vicinities for use in the syntheticreturn data computations from the spatial index based on the locationsof the non-targeted range points in the coordinate space.
 21. The methodof claim 16 wherein the detected ladar return data comprises intensitydata for the targeted range points, and wherein the synthetic returndata comprises synthetic intensity data for the non-targeted rangepoints.
 22. The method of claim 16 wherein the detected ladar returndata comprises range data for the targeted range points, and wherein thesynthetic return data comprises synthetic range data for thenon-targeted range points.
 23. The method of claim 16 furthercomprising: an image classification application processing thesynthetically-filled ladar frame to classify an object that is depictedin the synthetically-filled ladar frame.
 24. (canceled)
 25. The methodof claim 16 wherein the rolling scan skips two lines in a field of viewfor the ladar frame in ascending and descending shot schedules.
 26. Themethod of claim 16 further comprising: the processor estimating avelocity of an object depicted in the synthetically-filled ladar framesbased on comparative positions of the object in the synthetically-filledladar frames.
 27. The method of claim 26 wherein a synthetically-filledladar frame corresponding to an ascending scan of a field of view and asynthetically-filled ladar frame corresponding to a descending scan of afield of view are used by the processor for the velocity estimation. 28.The method of claim 26 wherein the object exhibits a size that is atleast as large as a skip length for the rolling scan.
 29. The method ofclaim 16 wherein the ladar system includes a ladar transmitter thattransmits ladar pulse shots into the coordinate space via a plurality ofscanning mirrors, and wherein the method further comprises: a processordynamically scheduling a plurality of ladar pulse shots for the ladartransmitter to target a plurality of range points in the coordinatespace; the ladar transmitter controllably scanning the mirrors to targetthe ladar transmitter at the targeted range points in accordance withthe scheduled ladar pulse shots; and the ladar transmitter transmittinga plurality of ladar pulse shots toward the targeted range points viathe scanning mirrors.
 30. The method of claim 29 wherein the dynamicallyscheduling step comprises a processor performing the dynamic schedulingon a shot-by-shot basis.